Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Ansible plugin now saves output and produces prepare/finish result#3738

Merged
happz merged 6 commits into
mainfrom
prepare-finish-ansible-results
Jul 15, 2025
Merged

Ansible plugin now saves output and produces prepare/finish result#3738
happz merged 6 commits into
mainfrom
prepare-finish-ansible-results

Conversation

@happz
Copy link
Copy Markdown
Contributor

@happz happz commented May 14, 2025

Pull Request Checklist

  • implement the feature
  • extend the test coverage

@happz happz added the step | prepare Stuff related to the prepare step label May 14, 2025
@happz happz added this to planning May 14, 2025
@happz happz added step | finish Stuff related to the finish step plugin | ansible The ansible plugins area | results Related to how tmt stores and shares results labels May 14, 2025
@github-project-automation github-project-automation Bot moved this to backlog in planning May 14, 2025
@happz happz moved this from backlog to review in planning May 14, 2025
@psss psss added this to the 1.53 milestone Jul 3, 2025
@psss psss requested a review from mcasquer July 3, 2025 12:05
@psss psss force-pushed the prepare-finish-ansible-results branch from 6cf5b15 to 28d7b93 Compare July 3, 2025 15:28
@psss psss requested review from psss, tcornell-bus and therazix July 3, 2025 15:31
@happz happz force-pushed the prepare-finish-ansible-results branch from 28d7b93 to ef718d4 Compare July 3, 2025 19:21
@happz happz added the ci | full test Pull request is ready for the full test execution label Jul 14, 2025
@happz happz force-pushed the prepare-finish-ansible-results branch from 5b70831 to 99a5f56 Compare July 14, 2025 11:47
@psss
Copy link
Copy Markdown
Member

psss commented Jul 14, 2025

Saving the ansible output seems quite useful, could be an interesting change for users, would it possibly deserve a short release note?

@happz
Copy link
Copy Markdown
Contributor Author

happz commented Jul 14, 2025

Saving the ansible output seems quite useful, could be an interesting change for users, would it possibly deserve a short release note?

Possibly. It can be argued it would be interesting, on the other hand, tmt is still pretty silent about these logs as they and their paths are not shown, unlike test logs. So my instict is to not bother user with something they would need to dig into a workdir to find it, and wait with the release to the moment prepare results are reported on console in the same way as test logs are. TL;DR, no preference, both ways have their pros and cons.

Comment thread tmt/steps/finish/__init__.py
@psss
Copy link
Copy Markdown
Member

psss commented Jul 14, 2025

So my instict is to not bother user with something they would need to dig into a workdir to find it, and wait with the release to the moment prepare results are reported on console in the same way as test logs are.

Understand the point. Would that possibly be just a single logger.verbose() or more complicated? If simple enough I'd include it right away and mention in the release notes. If not, let's keep the release not for later.

@happz
Copy link
Copy Markdown
Contributor Author

happz commented Jul 14, 2025

So my instict is to not bother user with something they would need to dig into a workdir to find it, and wait with the release to the moment prepare results are reported on console in the same way as test logs are.

Understand the point. Would that possibly be just a single logger.verbose() or more complicated? If simple enough I'd include it right away and mention in the release notes. If not, let's keep the release not for later.

More complicated, at least in my vision: I would like to reuse the display plugin, if it's available, and task it with reporting the results:

$ tmt run -i /tmp/tmp.ABoXAKPXIA --scratch -av provision -h local report -h display -vvv
/tmp/tmp.ABoXAKPXIA
warn: User is feeling safe.
Found 1 plan.

/plan
    discover
        how: shell
        summary: 1 test selected
            /script-00
    provision
        queued provision.provision task #1: default-0
        
        provision.provision task #1: default-0
        how: local
        primary address: localhost
        multihost name: default-0
        arch: x86_64
        distro: Fedora Linux 41 (Workstation Edition)
        kernel: 6.15.4-100.fc41.x86_64
        package manager: dnf5
        selinux: yes
        systemd: yes
        is superuser: no
        is_container: no
    
        summary: 1 guest provisioned
    prepare
        queued push task #1: push to default-0
        
        push task #1: push to default-0
    
        queued prepare task #1: essential-requires on default-0
        
        prepare task #1: essential-requires on default-0
        how: install
        summary: Install essential required packages
        name: essential-requires
        where: default-0
        package: 3 packages requested
            /usr/bin/awk
            /usr/bin/flock
            /usr/bin/python3
    
        queued pull task #1: pull from default-0
        
        pull task #1: pull from default-0
    
        summary: 1 preparation applied
    execute
        queued execute task #1: default-0 on default-0
        
        execute task #1: default-0 on default-0
        how: tmt
            00:00:00 pass /script-00 (on default-0) [1/1]

    
        summary: 1 test executed
    report
        how: display
        order: 50
            00:00:00 pass /script-00
                logs (with content):
                    output.txt (/tmp/tmp.ABoXAKPXIA/plan/execute/data/guest/default-0/script-00-1/output.txt)
                        content: fake
                    failures.yaml (/tmp/tmp.ABoXAKPXIA/plan/execute/data/guest/default-0/script-00-1/failures.yaml)
        summary: 1 test passed
    finish
        queued finish task #1: State before is valid (no file) on default-0
        queued finish task #2: Ansible we want to test on default-0
        queued finish task #3: State after is as expected (file created) on default-0
        queued finish task #4: Create a file that is pulled during the finish stage on default-0
        
        finish task #1: State before is valid (no file) on default-0
        how: shell
        name: State before is valid (no file)
        overview: 1 script found
        script: bash -xc "! [ -f /tmp/finished ]"
        
        finish task #2: Ansible we want to test on default-0
        how: ansible
        name: Ansible we want to test
        playbook: playbook.yml
        ok: 1 task
        changed: 1 task
        
        finish task #3: State after is as expected (file created) on default-0
        how: shell
        name: State after is as expected (file created)
        overview: 1 script found
        script: bash -xc "[ -f /tmp/finished ]"
        
        finish task #4: Create a file that is pulled during the finish stage on default-0
        how: shell
        name: Create a file that is pulled during the finish stage
        overview: 1 script found
        script: touch /tmp/tmp.ABoXAKPXIA/plan/data/my_file.txt
    
        ..:..:.. pass Ansible we want to test / playbook.yml
            logs (with content):
                output.txt (/tmp/tmp.ABoXAKPXIA/plan/finish/Ansible-we-want-to-test/0/default-0/output.txt)
                    content: ## Ansible we want to test / playbook.yml
                    content: 
                    content: # exit code: finished successfully
                    content: 
                    content: # stdout (16 lines)
                    content: # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                    content: PLAY [A simple playbook in finish phase] ***************************************
                    content: 
                    content: TASK [Gathering Facts] *********************************************************
                    content: ok: [localhost]
                    content: 
                    content: TASK [Create a simple test file] ***********************************************
                    content: changed: [localhost]
                    content: 
                    content: TASK [Make sure a simple environment variable is respected] ********************
                    content: ok: [localhost]
                    content: 
                    content: TASK [Make sure variables with spaces are correctly handed over] ***************
                    content: ok: [localhost]
                    content: 
                    content: PLAY RECAP *********************************************************************
                    content: localhost                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
                    content: # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
        queued pull task #1: pull from default-0
        
        pull task #1: pull from default-0
    
        summary: 4 tasks completed

total: 1 test passed

Note how display runs not just under report, that's expected, but also under finish where it's given the list of results collected from finish plugins. There is just one, shell plugin does not provide results yet, but that's something I would like to see. Or extend the report plugin to report on results from prepare, execute and finish, but report runs before finish...

Anyway, what I'd like to see is not a oneliner, although not complex either, the raw patch I have has 160 lines.

@happz
Copy link
Copy Markdown
Contributor Author

happz commented Jul 14, 2025

@coderabbitai full review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jul 14, 2025

✅ Actions performed

Full review triggered.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jul 14, 2025

📝 Walkthrough

Walkthrough

The changes introduce a new PluginOutcome class to standardize plugin result and exception reporting across the prepare and finish steps. All relevant plugin and step methods are refactored to use PluginOutcome instead of lists of PhaseResult. Exception handling and result aggregation logic is updated accordingly. Type annotations for plugin execution and asynchronous task management are aligned with these changes. The test script is enhanced to validate the new results format.

Changes

File(s) Change Summary
tmt/steps/__init__.py Added PluginOutcome class; updated PluginTask.run_on_guest to return plugin results; adjusted imports and type hints.
tmt/steps/prepare/__init__.py, tmt/steps/prepare/ansible.py, tmt/steps/prepare/distgit.py, tmt/steps/prepare/feature/__init__.py, tmt/steps/prepare/install.py, tmt/steps/prepare/shell.py Refactored all PreparePlugin and plugin go() methods to use PluginOutcome; updated outcome/result handling and error reporting.
tmt/steps/finish/__init__.py, tmt/steps/finish/shell.py Refactored FinishPlugin and related code to use PluginOutcome; updated result and exception aggregation and error handling.
tmt/queue.py Updated type annotations for task results and futures to use generic result types instead of None.
tmt/steps/provision/__init__.py Changed Guest.ansible method to return CommandOutput instead of None.
tests/finish/ansible/test.sh Enhanced test to check for the existence and content of the new results.yaml file.

Sequence Diagram(s)

sequenceDiagram
    participant Step as Prepare/Finish Step
    participant Plugin as Plugin (Prepare/Finish)
    participant Guest
    participant Logger

    Step->>Plugin: go(guest, environment, logger)
    Plugin->>Guest: (e.g., run playbook, install, etc.)
    alt Success
        Plugin-->>Step: PluginOutcome (results, exceptions=[])
    else Exception
        Plugin-->>Step: PluginOutcome (results, exceptions=[...])
    end
    Step->>Logger: Log results and exceptions
    Step->>Step: Aggregate results and exceptions
    alt Exceptions present
        Step->>Step: Raise Error with collected exceptions
    else
        Step->>Step: Save results
    end
Loading
sequenceDiagram
    participant MultiGuestTask
    participant Guest
    participant Logger

    MultiGuestTask->>Guest: run_on_guest(guest, logger)
    alt Success
        Guest-->>MultiGuestTask: TaskResultT (e.g., PluginOutcome)
    else Exception
        Guest-->>MultiGuestTask: Exception raised, captured in PluginOutcome
    end
    MultiGuestTask->>MultiGuestTask: Store result in futures[Future[TaskResultT]]
Loading

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai auto-generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
tmt/steps/finish/__init__.py (1)

230-236: Exception handling logic is correct.

The results are properly added on line 230 before handling exceptions and continuing. The past review concern has been adequately addressed.

🧹 Nitpick comments (2)
tmt/steps/provision/__init__.py (1)

1631-1631: Update the docstring to document the return value.

The return type annotation was correctly changed from None to tmt.utils.CommandOutput, but the method's docstring should be updated to document what is returned.

        :param silent: if set, logging of steps taken by this function would be
            reduced.
+        :returns: command output from the executed Ansible playbook, including
+            stdout and stderr.
        """
tmt/steps/prepare/__init__.py (1)

486-486: Address the TODO comment for error message improvement

Consider providing a more descriptive error message that includes context about which prepare plugins failed.

Would you like me to suggest a more informative error message or open an issue to track this improvement?

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e78711c and 6d3d0f7.

📒 Files selected for processing (12)
  • tests/finish/ansible/test.sh (1 hunks)
  • tmt/queue.py (2 hunks)
  • tmt/steps/__init__.py (4 hunks)
  • tmt/steps/finish/__init__.py (6 hunks)
  • tmt/steps/finish/shell.py (2 hunks)
  • tmt/steps/prepare/__init__.py (7 hunks)
  • tmt/steps/prepare/ansible.py (4 hunks)
  • tmt/steps/prepare/distgit.py (3 hunks)
  • tmt/steps/prepare/feature/__init__.py (2 hunks)
  • tmt/steps/prepare/install.py (2 hunks)
  • tmt/steps/prepare/shell.py (2 hunks)
  • tmt/steps/provision/__init__.py (2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (3)
tmt/steps/provision/__init__.py (1)
tmt/utils/__init__.py (2)
  • CommandOutput (1031-1033)
  • output (2444-2456)
tmt/steps/finish/shell.py (3)
tmt/steps/__init__.py (7)
  • PluginOutcome (1430-1439)
  • go (1268-1282)
  • go (2003-2008)
  • go (2016-2027)
  • go (2106-2107)
  • go (2176-2187)
  • go (2298-2304)
tmt/steps/finish/__init__.py (2)
  • go (73-82)
  • go (140-279)
tmt/steps/prepare/shell.py (1)
  • go (62-123)
tmt/steps/prepare/__init__.py (5)
tmt/result.py (2)
  • PhaseResult (297-300)
  • ResultOutcome (24-64)
tmt/base.py (1)
  • steps (2542-2553)
tmt/steps/__init__.py (11)
  • Action (2030-2107)
  • PhaseQueue (2661-2677)
  • PluginOutcome (1430-1439)
  • Plugin (2011-2027)
  • PluginTask (2614-2658)
  • run (2609-2610)
  • name (2606-2607)
  • name (2654-2655)
  • name (2691-2692)
  • name (2719-2720)
  • _save_results (850-861)
tmt/utils/__init__.py (8)
  • fail (2027-2032)
  • run (962-976)
  • run (995-996)
  • run (1150-1410)
  • run (2052-2104)
  • name (1578-1579)
  • name (1582-1588)
  • PrepareError (2560-2563)
tmt/log.py (1)
  • fail (871-878)
🔇 Additional comments (33)
tests/finish/ansible/test.sh (1)

56-60: LGTM! Well-structured test for the new results format.

The test enhancement properly validates the new structured results format produced by the finish step. The use of yq to parse and validate the YAML content structure is appropriate, and the expected output format aligns with the PluginOutcome refactoring.

tmt/steps/finish/shell.py (3)

73-73: LGTM! Type annotation updated correctly.

The return type change from list[PhaseResult] to tmt.steps.PluginOutcome aligns with the broader refactoring to standardize plugin return types using the PluginOutcome container class.


78-78: LGTM! Variable naming improved.

The variable rename from results to outcome is consistent with the new return type and improves code readability.


111-111: LGTM! Return statement updated consistently.

The return statement correctly returns the outcome variable, maintaining consistency with the type annotation and variable naming changes.

tmt/steps/prepare/shell.py (3)

68-68: LGTM! Type annotation updated correctly.

The return type change from list[PhaseResult] to tmt.steps.PluginOutcome aligns with the broader refactoring to standardize plugin return types using the PluginOutcome container class.


73-73: LGTM! Variable naming improved.

The variable rename from results to outcome is consistent with the new return type and improves code readability.


123-123: LGTM! Return statement updated consistently.

The return statement correctly returns the outcome variable, maintaining consistency with the type annotation and variable naming changes.

tmt/steps/provision/__init__.py (1)

1662-1662: Good change - method now returns the command output.

This aligns perfectly with the PR objective of making the Ansible plugin save its output. The returned CommandOutput object contains both stdout and stderr from the playbook execution, which enables downstream processing and logging.

tmt/steps/prepare/feature/__init__.py (3)

350-350: LGTM! Return type annotation updated to use PluginOutcome.

This change aligns with the broader refactoring to standardize plugin return types across the codebase.


355-355: Good practice: Variable renamed to match the new return type.

The renaming from results to outcome improves code clarity and consistency with the new PluginOutcome return type.


359-359: Return statements correctly updated.

All return statements have been properly updated to use the renamed outcome variable.

Also applies to: 384-384

tmt/queue.py (2)

200-200: LGTM! Return type annotation updated to reflect actual return value.

The method now correctly indicates it returns a TaskResultT instead of None, which aligns with the actual behavior where task results are returned.


232-232: Type annotation correctly updated for futures dictionary.

The futures dictionary type has been properly updated to reflect that futures hold results of type TaskResultT instead of None.

tmt/steps/prepare/install.py (3)

892-892: LGTM! Return type annotation updated to use PluginOutcome.

This change is consistent with the broader refactoring to standardize plugin return types across all prepare plugins.


897-897: Good practice: Variable renamed to match the new return type.

The renaming from results to outcome improves code clarity and consistency with the new PluginOutcome return type.


901-901: Return statements correctly updated.

All return statements have been properly updated to use the renamed outcome variable, maintaining the existing logic flow.

Also applies to: 995-995

tmt/steps/prepare/distgit.py (4)

190-190: LGTM! Return type annotation updated to use PluginOutcome.

This change is consistent with the broader refactoring to standardize plugin return types across all prepare plugins.


195-195: Good practice: Variable renamed to match the new return type.

The renaming from results to outcome improves code clarity and consistency with the new PluginOutcome return type.


281-282: Excellent practice: Variable renamed to avoid naming conflict.

Renaming the guest command execution result from outcome to output prevents confusion with the outer outcome variable. This demonstrates good attention to detail in the refactoring effort.


347-347: Return statement correctly updated.

The return statement has been properly updated to use the renamed outcome variable.

tmt/steps/__init__.py (2)

1429-1441: Well-designed plugin outcome container!

The PluginOutcome class provides a clean way to bundle results and exceptions from plugin executions. Good use of simple_field with default factories to avoid mutable default issues.


2657-2658: Return type correctly updated to support plugin outcomes.

The change from returning None to returning the plugin's result aligns perfectly with the new PluginOutcome pattern.

tmt/steps/finish/__init__.py (2)

36-36: Return type correctly updated to PluginOutcome.

The change aligns with the new unified plugin result handling pattern across all step plugins.

Also applies to: 79-82


189-197: Excellent exception aggregation pattern!

The approach of collecting all exceptions and raising a single GeneralError with all exceptions as causes provides comprehensive error reporting while allowing all plugins to execute.

Also applies to: 240-244

tmt/steps/prepare/ansible.py (3)

173-179: Smart approach to preserve playbook log directories!

Using numbered directories and dynamically generating the preserved set based on playbook count ensures all logs are retained.


198-306: Excellent refactoring of playbook execution logic!

The structured approach with dedicated directories per playbook, comprehensive error handling, and proper result accumulation significantly improves debuggability and error reporting.


268-291: Robust error handling with appropriate logging!

The differentiation between RunError (with output logging) and generic exceptions (without logs) is well thought out. Early returns on errors prevent cascading failures.

tmt/steps/prepare/__init__.py (6)

2-2: LGTM!

The import additions are necessary for the type annotations used in the refactored code.


17-21: LGTM!

The imports are correctly added to support the new PluginOutcome based return values and error result creation.


48-48: LGTM!

The base class correctly specifies PluginOutcome as the return type, aligning with the new plugin result handling architecture.


96-111: LGTM!

The method signature and default implementation correctly use the new PluginOutcome return type.


416-416: LGTM!

The queue type parameters are correctly updated to handle PluginOutcome results.


433-441: LGTM!

The exception collection mechanism and helper function provide a clean way to aggregate and report multiple exceptions from different plugins.

Comment thread tmt/steps/prepare/__init__.py
@psss
Copy link
Copy Markdown
Member

psss commented Jul 14, 2025

Anyway, what I'd like to see is not a oneliner, although not complex either, the raw patch I have has 160 lines.

Ok, makes sense, let's handle that separately.

Copy link
Copy Markdown
Member

@psss psss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Nice improvement! Just one nitpick comment which can safely be ignored.

Comment thread tmt/steps/prepare/ansible.py Outdated
@psss psss moved this from review to merge in planning Jul 15, 2025
@psss psss force-pushed the prepare-finish-ansible-results branch from 63ee870 to 47d6853 Compare July 15, 2025 10:44
@happz
Copy link
Copy Markdown
Contributor Author

happz commented Jul 15, 2025

Unrelated error, merging.

@happz happz merged commit 39dd66c into main Jul 15, 2025
17 of 18 checks passed
@happz happz deleted the prepare-finish-ansible-results branch July 15, 2025 13:00
@github-project-automation github-project-automation Bot moved this from merge to done in planning Jul 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area | results Related to how tmt stores and shares results ci | full test Pull request is ready for the full test execution plugin | ansible The ansible plugins step | finish Stuff related to the finish step step | prepare Stuff related to the prepare step

Projects

Status: done

Development

Successfully merging this pull request may close these issues.

3 participants