RFC: create state.exists #45730

mchugh19 · 2018-01-27T22:32:45Z

What does this PR do?

A hack which duplicates the state.show_sls to return True or False. Because states can live in gitfs, a simple file.exists isn't enough to support this need.

See this top file where states are assigned from a roles grain.
States are assigned to hosts, based on the roles listed in the roles grain. Without any other check, any roles in the grain but missing accompanying state files, will cause highstate to fail. By wrapping the role assignment in top with a check to validate it exists, these failures can be avoided.

# cat top.sls
base:
  '*':
    - common

  {% set roles = salt['grains.get']('roles',[]) -%}
  {% for role in roles -%}
  {% if salt['state.exists']('roles.{0}'.format(role)) -%}
  'roles:{{ role }}':
    - match: grain
    - roles.{{ role }}
  {% endif -%}
  {% endfor -%}

# salt-call state.show_top
[ERROR   ] Template was specified incorrectly: False
local:
    ----------
    base:
        - common
# touch roles/saltmaster.sls
# salt-call state.show_top
local:
    ----------
    base:
        - common
        - roles.saltmaster

What issues does this PR fix or reference?

Intent was to cover use cases such as #16687
It would be better to add functionality like ignore_missing to states, but how to do so was not immediately obvious.

Tests written?

No

Commits signed with GPG?

No

terminalmage

Duplicating all of the code in show_sls is not necessary here. If there were errors, a list is returned, if not, a dict is returned. Therefore, this function can re-written like so:

return isinstance(
    show_sls(mods, test=test, queue=queue, **kwargs),
    dict
)

Also, I would prefer the name of the function be changed to sls_exists. This leaves open the option later on of adding a function to check if a given ID exists within a specified SLS file, and naming that function id_exists.

…ists

Add unit tests

mchugh19 · 2018-02-10T08:30:31Z

Fixed and renamed to sls_exists. Also added id_exists, and unit tests for both.

terminalmage

A few docstring changes need to be made, and the id_exists function doesn't quite do what I was describing earlier. I've left an explanation and some example code to get you pointed in the right direction.

terminalmage · 2018-02-13T05:32:33Z

salt/modules/state.py

+def sls_exists(mods, test=None, queue=False, **kwargs):
+    '''
+    Tests for the existance the of a specific sls or list of sls files on the
+    master. Similar to show_sls, rather than returning state details, returns


This can be improved by making show_sls a direct link to the docs for that function. For example, instead of "show_sls", you can do this:

:py:func:`state.show_sls <salt.modules.state.show_sls>`

When our docs are built, state.show_sls will link directly to the docs for that function.

terminalmage · 2018-02-13T05:51:10Z

salt/modules/state.py

+    for state_id in ids:
+        if state_id not in state_ids:
+            return False
+    return True


I think there was a misunderstanding of my earlier comment. When I referenced the concept of an "ID", I was talking about an individual state within an SLS file. The ID would be the top-level declaration for a given state. For instance, given the below SLS:

run_date_cmd: cmd.run: - name: date zsh: pkg.installed

The IDs would be run_date_cmd and zsh.

Though I haven't tested this at all, the following code should work (or at least get you most of the way there):

if isinstance(ids, six.string_types): ids = ids.split(',') ids = set(ids) sls_ids = set(x['__id__'] for x in show_low_sls(mods, test=test, queue=queue, **kwargs)) return ids.issubset(sls_ids)

state.show_low_sls gives you a list of what we call the "low chunks", each chunk being a dictionary representing a single state, so if all of the passed IDs exist, then they should be a subset of all of the IDs in the specified SLS files.

Note that you'd need to rewrite the test as well.

I believe the described functionality is what was implemented. The line
state_ids = show_sls(mods, test=test, queue=queue, **kwargs).keys()
causes the state to be rendered and the top level key names gathered. Thus, the state_ids variable should be a list of the state IDs.

Sounds like using _id_ key from lowstate might be a superior method, so I'll take a look.

terminalmage · 2018-02-13T05:53:23Z

salt/modules/state.py

+    .. versionadded:: Fluorine
+
+    saltenv
+        Specify a salt fileserver environment to be used when applying states


We're not actually applying states here, it would be more accurate to say:

Specify a salt fileserver environment from which to look for the SLS files specified in the ``mods`` argument.

terminalmage · 2018-02-13T05:57:18Z

salt/modules/state.py

+    '''
+    Tests the for a specific id or list of ids within a given state on the
+    master. Similar to sls_exists, returns True or False. The default
+    environment is ``base``, use ``saltenv`` to specify a different environment.


This would be better worded as:

Tests for the existence of a specific ID or list of IDs within the specified SLS file(s). Similar to :py:func:`state.sls_exists <salt.modules.state.sls_exists>`, returns True or False. The default environment is base``, use ``saltenv`` to specify a different environment.

terminalmage · 2018-02-13T05:58:39Z

salt/modules/state.py

+    .. versionadded:: Fluorine
+
+    saltenv
+        Specify a salt fileserver environment to be used when applying states


We're not actually applying states here, it would be more accurate to say:

Specify a salt fileserver environment from which to look for the SLS files specified in the ``mods`` argument.

terminalmage · 2018-02-13T06:05:36Z

salt/modules/state.py


+def sls_exists(mods, test=None, queue=False, **kwargs):
+    '''
+    Tests for the existance the of a specific sls or list of sls files on the


SLS should be all-capitals when referring to an SLS file.

Improve id_exists by using show_low_sls and checking __id__ keys.

mchugh19 · 2018-02-17T14:29:15Z

Thanks @terminalmage. Suggestions/corrections applied.

mchugh19 · 2018-02-28T12:52:57Z

I think all of the requested updates have been submitted. Is there anything outstanding?

terminalmage

This looks good, thanks for your patience as we've worked through this.

isbm · 2018-02-28T22:30:48Z

salt/modules/state.py

+        salt '*' state.id_exists create_myfile,update_template filestate saltenv=dev
+    '''
+    if isinstance(ids, six.string_types):
+        ids = ids.split(',')


Please use salt.utils.args.split_input instead of this:

def id_exists(... sls_ids = set(x['__id__'] for x ... return set(salt.utils.args.split_input(ids)).issubset(sls_ids)

And it would be nice to others not to use one-char variables.

I don't mind the 1-char in a comprehension expression, to be honest. I don't think that needs to change.

@isbm is right about the split_input, we do have a helper for this. It could be done as

ids = salt.utils.args.split_input(ids)

…_input(variable)

mchugh19 · 2018-03-01T08:13:47Z

Updated to use salt.utils.args.split_input. While there, I also updated the other uses of variable.split(',') in the file to keep everything consistent. Is that alright?

isbm · 2018-03-01T09:21:30Z

Updated to use salt.utils.args.split_input. While there, I also updated the other uses of variable.split(',') in the file to keep everything consistent. Is that alright?

Yes, this is alright. Almost. 😉 You just don't need to extra-check if isinstance... is string. Just split directly (and so remove all those checks by the way).

Thanks!

mchugh19 · 2018-03-01T10:00:52Z

Ah, that makes sense. Done!

isbm

@mchugh19 @rallytime @terminalmage LGTM! 👍

Christian McHugh and others added 2 commits January 27, 2018 22:15

create state.exists

5a94161

Merge branch 'develop' into state.exists

e7039e8

rallytime requested a review from a team February 8, 2018 14:33

terminalmage suggested changes Feb 8, 2018

View reviewed changes

Christian McHugh added 3 commits February 9, 2018 18:08

refine state_exists and add id_exists

9993a62

Merge branch 'state.exists' of github.com:mchugh19/salt into state.ex…

127497e

…ists

Handle invalid states on id_exists

28fb5a6

Add unit tests

rallytime requested a review from a team February 12, 2018 16:46

terminalmage suggested changes Feb 13, 2018

View reviewed changes

Fix wording

cee4333

Improve id_exists by using show_low_sls and checking __id__ keys.

rallytime requested review from terminalmage and a team February 28, 2018 14:49

terminalmage approved these changes Feb 28, 2018

View reviewed changes

isbm suggested changes Feb 28, 2018

View reviewed changes

convert instances of variable.split(',') to use salt.utils.args.split…

d4c5c9a

…_input(variable)

properly use salt.utils.args.split and remove wrapping logic

69386df

isbm approved these changes Mar 1, 2018

View reviewed changes

terminalmage approved these changes Mar 3, 2018

View reviewed changes

Merge branch 'develop' into state.exists

8ec1c66

rallytime merged commit c99dbd8 into saltstack:develop Mar 5, 2018

mchugh19 mentioned this pull request Aug 28, 2018

Include State only if it exists #20010

Closed

RFC: create state.exists #45730

RFC: create state.exists #45730

Uh oh!

Conversation

mchugh19 commented Jan 27, 2018

What does this PR do?

What issues does this PR fix or reference?

Tests written?

Commits signed with GPG?

Uh oh!

terminalmage left a comment

Choose a reason for hiding this comment

Uh oh!

mchugh19 commented Feb 10, 2018

Uh oh!

terminalmage left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchugh19 Feb 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchugh19 commented Feb 17, 2018

Uh oh!

mchugh19 commented Feb 28, 2018

Uh oh!

terminalmage left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchugh19 commented Mar 1, 2018

Uh oh!

isbm commented Mar 1, 2018

Uh oh!

mchugh19 commented Mar 1, 2018

Uh oh!

isbm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mchugh19 Feb 17, 2018 •

edited

Loading