Script CI/CD Validator

A robust framework for testing, safety, and security of PowerShell, Python, and Shell scripts through automated validation in both local and CI/CD environments.

---

Philosophy & Goals

• Safety: Prevent dangerous or destructive code from being merged.
• Security: Proactively detect vulnerabilities and unsafe patterns.
• Quality: Enforce best practices, style, and maintainability across all scripts.
• Transparency: Provide actionable, clear feedback to contributors at every stage.

---

How Validation Works

1. Local Testing (Pre-Commit)

• Makefile orchestrates all checks for every supported language.
• Pipenv manages Python tool dependencies for reproducibility.
• Feedback: Each tool prints clear, actionable output. All issues are shown, even if some checks fail.
• How to Run:

  make setup      # Install dependencies (first time)
  make lint-all   # Run all checks

• What You'll See:
  • Section headers for each tool/language
  • Diffs for files needing formatting (e.g., Ruff, shfmt)
  • Linter/security warnings with file names and line numbers
  • All issues for all scripts, not just the first failure

2. Automated CI/CD (GitHub Actions)

• Every pull request triggers .github/workflows/script-validation.yml.
• The workflow runs the same checks as local testing:
  • Python: Ruff (lint/format), Pylint, Bandit
  • PowerShell: PSScriptAnalyzer
  • Shell: ShellCheck, shfmt
• Feedback: All output is visible in the PR’s Actions tab. Contributors can see exactly what failed and why.
• Blocking: PRs with unresolved issues cannot be merged.

---

Safety & Security Practices

• No hardcoded secrets: All scripts are scanned for credentials, tokens, and other sensitive data.
• No dangerous commands: Patterns like os.system('rm -rf /') or unsafe shell constructs are flagged.
• Security scanning:
  • Python: Bandit checks for common vulnerabilities (e.g., use of eval, unsafe subprocess calls).
  • PowerShell: PSScriptAnalyzer flags unsafe or deprecated patterns.
  • Shell: ShellCheck highlights dangerous or error-prone constructs.
• Custom rules: Config files (PSScriptAnalyzerSettings.psd1, .shellcheckrc, .pylintrc, pyproject.toml) enforce project-specific standards.

---

Local Testing: Step-by-Step

1. Install pipenv:

   pip install pipenv

2. Install dependencies:

   make setup

3. Run all validations:

   make lint-all

4. Review output:
   • Fix any issues shown in the output (formatting, lint, security, etc.).
   • Re-run make lint-all until all checks pass.

Sample Output:

========== PYTHON: Ruff (format check, with diff) ==========
--- tests/python/bad_script.py
+++ tests/python/bad_script.py
@@ ... diff output ...
Would reformat: tests/python/bad_script.py
========== SHELL: ShellCheck ==========
tests/shell/bad_script.sh: line 2: SC2086: Double quote to prevent globbing and word splitting.
...

---

Example Output for Each Script Type

Below are real examples of what you’ll see for each language when issues are detected. Use these as a reference to interpret and fix problems in your scripts.

Python

Ruff (format check, with diff):

========== PYTHON: Ruff (format check, with diff) ==========
--- tests/python/bad_script.py
+++ tests/python/bad_script.py
@@ -1,5 +1,8 @@
 # bad_script.py
 # This script violates several linting and security rules
 import os
-print('This line is too long and will trigger a lint error because it exceeds the maximum line length set in the configuration file, and also uses a print statement which is discouraged in some contexts.')
-os.system('rm -rf /')  # Dangerous pattern for Bandit
+
+print(
+    "This line is too long and will trigger a lint error because it exceeds the maximum line length set in the configuration file, and also uses a print statement which is discouraged in some contexts."
+)
+os.system("rm -rf /")  # Dangerous pattern for Bandit

Would reformat: tests/python/bad_script.py
1 file would be reformatted

Pylint:

========== PYTHON: Pylint ==========
************* Module bad_script
bad_script.py:5:0: C0301: Line too long (120/100) (line-too-long)
bad_script.py:6:0: W1510: Using subprocess without shell equals true is dangerous (subprocess-run-check)
...

Bandit:

========== PYTHON: Bandit ==========
>> Issue: [B602:subprocess_popen_with_shell_equals_true] subprocess call with shell=True identified, security issue.
   Severity: Medium   Confidence: High
   Location: tests/python/bad_script.py:6

os.system("rm -rf /")  # Dangerous pattern for Bandit
...

PowerShell

PSScriptAnalyzer:

========== POWERSHELL: PSScriptAnalyzer ==========

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
PSAvoidUsingWriteHost               Warning      bad_script.ps1 3  'Write-Host' is not recommended.
PSAvoidUsingPlainTextForPassword    Error        bad_script.ps1 5  Storing passwords as plain text is insecure.
...

Shell

ShellCheck:

========== SHELL: ShellCheck ==========
tests/shell/bad_script.sh:2:6: warning: Double quote to prevent globbing and word splitting. [SC2086]
tests/shell/bad_script.sh:4:1: note: Tips depend on target shell and yours is unknown. Add a shebang. [SC2148]
...

shfmt (format check, with diff):

========== SHELL: shfmt (format check, with diff) ==========
--- tests/shell/bad_script.sh
+++ tests/shell/bad_script.sh
@@ ... diff output ...
Would reformat: tests/shell/bad_script.sh
...

OSQuery

osqueryi (syntax check):

========== OSQUERY: Syntax Check ==========
Checking queries/bad_query.osquery...
Error: near "SELEC": syntax error
Syntax error in queries/bad_query.osquery

sqlfluff (style lint):

========== OSQUERY: SQLFluff Lint ==========
== [queries/bad_query.osquery] FAIL
L:   1 | P:   1 | L003 | Indentation not consistent with line #2
...

---

Contribution & Review Process

• Before PR:
  • Run make lint-all locally and fix all issues.
  • Ensure your script is safe, secure, and follows best practices.
• On PR:
  • GitHub Actions will validate your changes.
  • Reviewers and CODEOWNERS (if set) will be notified of issues.
  • PRs with unresolved issues cannot be merged.
• Documentation:
  • See CONTRIBUTING.md for detailed instructions and troubleshooting.

---

Tools & Configurations

• Python: Ruff, Pylint, Bandit
• PowerShell: PSScriptAnalyzer
• Shell: ShellCheck, shfmt
• OSQuery: osqueryi, sqlfluff
• Config files:
  • .pylintrc, pyproject.toml, PSScriptAnalyzerSettings.psd1, .shellcheckrc, .osquery
• Dependency management: pipenv
• Automation: Makefile, GitHub Actions

---

FAQ

• Q: What if my script fails a check?
  • Read the output, fix the reported issue, and re-run make lint-all.
• Q: How do I know if my script is secure?
  • All scripts are scanned for dangerous patterns and vulnerabilities. If Bandit, ShellCheck, or PSScriptAnalyzer flags something, address it or explain why it’s safe.
• Q: Can I bypass a check?
  • Only with a clear justification and approval from maintainers. Never bypass security checks lightly.

---

License

MIT License