Multiple issues with Saltclass variable expansion #48170
Description
Description of Issue/Question
I spent more than 2 days trying to migrate existing reclass tree to saltclass (introduced in #42349) and found multiple issues with saltclass variable expansion.
- Variable expansion doesn't work at all under Python 2.7
The reason is that checking for str is not enough, strings can also be unicode with 2.7: https://github.com/saltstack/salt/blob/2018.3.1/salt/utils/saltclass.py#L137
This feature definitely needs more unit tests. Jenkins builds for 2.7 can catch issues like this.
- Not all variables are expanded
Salt 2017.7, reclass:
% salt-call --id example.com pillar.data
local:
----------
__reclass__:
----------
applications:
- consul
classes:
- consul
- consul.agent
- site.mydc
environment:
base
nodename:
example.com
consul:
----------
agent:
----------
args:
- datacenter=mydc
- join=1.2.3.4
cur_version:
1.0.3
dc:
----------
name:
mydc
servers:
1.2.3.4
versions:
----------
1.0.3:
----------
source:
https://releases.hashicorp.com/consul/1.0.3/consul_1.0.3_linux_amd64.zip
source_hash:
sha256=4782e4662de8effe49e97c50b1a1233c03c0026881f6c004144cc3b73f446ec5
Salt 2018.3.1, saltclass
local:
----------
__saltclass__:
----------
classes:
- consul
- consul.agent
- site.mydc
environment:
nodename:
example.com
states:
- consul
consul:
----------
agent:
----------
args:
- datacenter=${consul:dc:name}
- join=${consul:dc:servers}
cur_version:
1.0.3
versions:
----------
1.0.3:
----------
source:
https://releases.hashicorp.com/consul/1.0.3/consul_1.0.3_linux_amd64.zip
source_hash:
sha256=4782e4662de8effe49e97c50b1a1233c03c0026881f6c004144cc3b73f446ec5
Minimal state tree (both for reclass and saltclass) to reproduce: example.zip
- Multiple variables on a single line probably won't be expanded
Variable parsing code tells me that things like one below won't work:
rules:
- '-A PREROUTING -s 1.2.3.4/32 -d ${ipv4:wan} -p tcp --dport 1234 ${_fw_:dnat} 5.6.7.8:1234'
It worked just fine with reclass.
- Missing variables are rendered as is, which is hard to debug and potentially dangerous
I spent hours trying to understand why Jinja rendering fails with Jinja variable 'str object' has no attribute 'get'. We use variable expansion for pillar subtrees and due to issue (2) it returned unexpanded variable as str instead of dict.
It is essential to hard fail on missing variables (and classes). If necessary, I will create a separate issue for this.
Any typo in a variable or class name should lead to failure with an appropriate error message; otherwise, it will eventually lead to silent failures in production. Failing hard is also useful for simple automated linting/testing.
So far I'm not ready to put Saltclass to production. Instead I fixed reclass to make it work with Salt 2018.3: madduck/reclass#83
Versions Report
salt --versions-report
Salt Version: Salt: 2018.3.1Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.15 (default, May 2 2018, 00:53:27)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 17.0.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.6
System Versions:
dist:
locale: UTF-8
machine: x86_64
release: 17.6.0
system: Darwin
version: 10.13.5 x86_64