what

• Add Google Cloud Storage (GCS) backend support to !terraform.state Atmos YAML function
• Implement performance optimizations (client caching, retry logic, extended timeouts)
• Create unified Google Cloud authentication system for consistency across GCP services
• Update documentation with GCS backend usage examples and authentication methods

why

The !terraform.state YAML function allows reading the outputs (remote state) of components in Atmos stack manifests directly from the configured Terraform/OpenTofu backends.

Previously, the !terraform.state YAML function only supported:

• local (Terraform and OpenTofu)
• s3 (Terraform and OpenTofu)

This PR adds support for:

• gcs (Google Cloud Storage - Terraform and OpenTofu)

With GCS backend support, users can now leverage the high-performance !terraform.state function instead of the slower !terraform.output or !store functions when using Google Cloud Storage for Terraform state storage.

Implementation Details

GCS Backend Features

• Full Authentication Support: JSON credentials, service account file paths, and Google Application Default Credentials (ADC)
• Service Account Impersonation: Support for impersonate_service_account configuration
• Performance Optimizations:
  • Client caching to avoid recreating GCS clients for repeated operations
  • Retry logic with exponential backoff (up to 3 attempts) for transient failures
  • Extended timeouts (30 seconds) to match S3 backend performance
• Robust Error Handling: Graceful handling of missing state files and detailed error context
• Resource Management: Proper cleanup and explicit resource management

Usage

The GCS backend works seamlessly with existing !terraform.state syntax:

# Get the `output` of the `component` in the current stack
subnet_id: !terraform.state vpc private_subnet_id

# Get the `output` of the `component` in the provided `stack` 
vpc_id: !terraform.state vpc dev-us-east-1 vpc_id

# Get complex outputs using YQ expressions
first_subnet: !terraform.state vpc .private_subnet_ids[0]

GCS Backend Configuration

The GCS backend supports all standard Terraform GCS backend configurations:

# atmos.yaml
components:
  terraform:
    backend_type: gcs
    backend:
      gcs:
        bucket: "my-terraform-state-bucket"
        prefix: "terraform/state"
        
        # Authentication options (choose one):
        
        # Option 1: JSON credentials content
        credentials: |
          {
            "type": "service_account",
            "project_id": "my-project",
            ...
          }
          
        # Option 2: Service account file path  
        credentials: "/path/to/service-account.json"
        
        # Option 3: Use Application Default Credentials (ADC)
        # (no credentials field needed - uses environment/metadata)
        
        # Optional: Service account impersonation
        impersonate_service_account: "[email protected]"

Performance Benefits

Compared to !terraform.output, the !terraform.state function with GCS backend:

• ✅ No Terraform execution - Reads state directly from GCS
• ✅ No provider initialization - Skips all module and provider setup
• ✅ No varfile generation - Bypasses Terraform configuration preparation
• ✅ Cached clients - Reuses GCS clients for multiple operations
• ✅ Parallel execution - Multiple state reads can happen concurrently

Testing

• Comprehensive Test Suite: 100% test coverage for all new functionality
• Mock Implementations: Complete interface-based testing for GCS operations
• Authentication Testing: Validates all credential types and authentication flows
• Error Scenario Coverage: Tests for missing files, network failures, and invalid configurations
• Caching Validation: Ensures client caching works correctly across operations
• Retry Logic Testing: Validates exponential backoff and failure recovery

Backward Compatibility

• ✅ No breaking changes to existing configurations
• ✅ Existing backends (local, s3) remain unchanged
• ✅ Same function syntax - no new parameters or options required
• ✅ Graceful fallbacks - continues to work with !terraform.output and !store functions

Files Changed

Core Implementation

• internal/terraform_backend/terraform_backend_gcs.go - GCS backend implementation
• internal/terraform_backend/terraform_backend_gcs_test.go - Comprehensive test suite
• internal/terraform_backend/terraform_backend_registry.go - Register GCS backend
• internal/terraform_backend/terraform_backend_utils.go - Updated error messages

Unified Authentication System

• internal/gcp/auth.go - New unified Google Cloud authentication (created)
• internal/gcp/auth_test.go - Authentication tests (created)
• pkg/store/google_secret_manager_store.go - Updated to use unified auth
• internal/gcp_utils/gcp_utils.go - Removed (replaced by unified auth)

Configuration & Documentation

• internal/exec/terraform_generate_backend.go - GCS backend validation
• website/docs/core-concepts/stacks/yaml-functions/terraform.state.mdx - Updated documentation
• errors/errors.go - Added GCS-specific error types
• go.mod - Added GCS storage dependency

Migration Guide

For users currently using !terraform.output or !store with GCS-stored state:

Before (slower)

# Using !terraform.output (requires Terraform execution)
vpc_id: !terraform.output vpc dev-us-east-1 vpc_id

# Using !store (requires separate state management)  
vpc_id: !store google-secret-manager dev/vpc/vpc_id

After (faster)

# Using !terraform.state (direct GCS state access)
vpc_id: !terraform.state vpc dev-us-east-1 vpc_id

Simply update your backend configuration to use gcs and replace function calls - no other changes needed!

Summary by CodeRabbit

• New Features

  • GCS-backed Terraform state support and unified Google Cloud authentication integration.

• Bug Fixes

  • Stricter backend config validation with clearer error responses and updated supported-backends messaging.

• Tests

  • Comprehensive unit tests added for GCS backend behavior and GCP authentication handling.

This PR addresses multiple authentication issues when using Atmos in containerized environments with mounted credential files:

1. Auth Pre-Hook Error Propagation - Terraform execution now properly aborts when authentication fails (e.g., Ctrl+C during SSO)
2. AWS Credential Loading Strategy - New LoadAtmosManagedAWSConfig() function provides proper isolation while preserving Atmos-managed profile selection
3. Noop Keyring Validation - Container auth now properly isolated from external environment variables
4. Whoami with Noop Keyring - atmos auth whoami now works in containerized environments
5. Test Coverage - Added test to verify auth errors properly abort execution

Changes

1. Auth Pre-Hook Error Propagation (internal/exec/terraform.go:236)

• Problem: Errors from auth pre-hook were logged but not returned, causing terraform execution to continue even when authentication failed (e.g., user presses Ctrl+C during SSO)
• Fix: Added return err after logging auth pre-hook errors
• Impact: Terraform commands now properly abort on auth failures

2. AWS Credential Loading Strategy (pkg/auth/cloud/aws/env.go)

• Problem: SDK's default config loading allowed IMDS access and was affected by external AWS_PROFILE, causing conflicts in containers
• Solution: Created LoadAtmosManagedAWSConfig() function that:
  • Clears credential env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  • Preserves profile/path vars (AWS_PROFILE, AWS_SHARED_CREDENTIALS_FILE, AWS_CONFIG_FILE)
  • Allows SDK to load from Atmos-managed credential files
• Impact: Proper isolation while still using Atmos-managed profiles

3. Noop Keyring Credential Validation (pkg/auth/credentials/keyring_noop.go)

• Problem: Used unrestricted config.LoadDefaultConfig() which allowed IMDS access and was affected by external AWS_PROFILE
• Fix: Changed to use LoadAtmosManagedAWSConfig()
• Impact: Container auth now properly isolated from external env vars

4. Whoami with Noop Keyring (pkg/auth/manager.go)

• Problem: Whoami() expected credentials from keyring, but noop keyring returns ErrCredentialsNotFound by design
• Fix: Added check for ErrCredentialsNotFound and fallback to buildWhoamiInfoFromEnvironment()
• Impact: atmos auth whoami now works in containerized environments

5. Test Coverage (internal/exec/terraform_test.go)

• Added TestExecuteTerraform_AuthPreHookErrorPropagation to verify auth errors properly abort execution
• Test validates that terraform doesn't continue on auth failure
• Updated test fixture to include required name_pattern configuration

Technical Details

The key insight is that Atmos sets AWS_PROFILE=identity-name (in pkg/auth/cloud/aws/setup.go:59) but the previous isolation approach cleared ALL AWS env vars including AWS_PROFILE. This caused the SDK to look for a non-existent [default] section.

The new LoadAtmosManagedAWSConfig preserves `AWS_...

what

• Add Google Cloud Storage (GCS) backend support to !terraform.state Atmos YAML function
• Implement performance optimizations (client caching, retry logic, extended timeouts)
• Create unified Google Cloud authentication system for consistency across GCP services
• Update documentation with GCS backend usage examples and authentication methods

why

The !terraform.state YAML function allows reading the outputs (remote state) of components in Atmos stack manifests directly from the configured Terraform/OpenTofu backends.

Previously, the !terraform.state YAML function only supported:

• local (Terraform and OpenTofu)
• s3 (Terraform and OpenTofu)

This PR adds support for:

• gcs (Google Cloud Storage - Terraform and OpenTofu)

With GCS backend support, users can now leverage the high-performance !terraform.state function instead of the slower !terraform.output or !store functions when using Google Cloud Storage for Terraform state storage.

Implementation Details

GCS Backend Features

• Full Authentication Support: JSON credentials, service account file paths, and Google Application Default Credentials (ADC)
• Service Account Impersonation: Support for impersonate_service_account configuration
• Performance Optimizations:
  • Client caching to avoid recreating GCS clients for repeated operations
  • Retry logic with exponential backoff (up to 3 attempts) for transient failures
  • Extended timeouts (30 seconds) to match S3 backend performance
• Robust Error Handling: Graceful handling of missing state files and detailed error context
• Resource Management: Proper cleanup and explicit resource management

Usage

The GCS backend works seamlessly with existing !terraform.state syntax:

# Get the `output` of the `component` in the current stack
subnet_id: !terraform.state vpc private_subnet_id

# Get the `output` of the `component` in the provided `stack` 
vpc_id: !terraform.state vpc dev-us-east-1 vpc_id

# Get complex outputs using YQ expressions
first_subnet: !terraform.state vpc .private_subnet_ids[0]

GCS Backend Configuration

The GCS backend supports all standard Terraform GCS backend configurations:

# atmos.yaml
components:
  terraform:
    backend_type: gcs
    backend:
      gcs:
        bucket: "my-terraform-state-bucket"
        prefix: "terraform/state"
        
        # Authentication options (choose one):
        
        # Option 1: JSON credentials content
        credentials: |
          {
            "type": "service_account",
            "project_id": "my-project",
            ...
          }
          
        # Option 2: Service account file path  
        credentials: "/path/to/service-account.json"
        
        # Option 3: Use Application Default Credentials (ADC)
        # (no credentials field needed - uses environment/metadata)
        
        # Optional: Service account impersonation
        impersonate_service_account: "[email protected]"

Performance Benefits

Compared to !terraform.output, the !terraform.state function with GCS backend:

• ✅ No Terraform execution - Reads state directly from GCS
• ✅ No provider initialization - Skips all module and provider setup
• ✅ No varfile generation - Bypasses Terraform configuration preparation
• ✅ Cached clients - Reuses GCS clients for multiple operations
• ✅ Parallel execution - Multiple state reads can happen concurrently

Testing

• Comprehensive Test Suite: 100% test coverage for all new functionality
• Mock Implementations: Complete interface-based testing for GCS operations
• Authentication Testing: Validates all credential types and authentication flows
• Error Scenario Coverage: Tests for missing files, network failures, and invalid configurations
• Caching Validation: Ensures client caching works correctly across operations
• Retry Logic Testing: Validates exponential backoff and failure recovery

Backward Compatibility

• ✅ No breaking changes to existing configurations
• ✅ Existing backends (local, s3) remain unchanged
• ✅ Same function syntax - no new parameters or options required
• ✅ Graceful fallbacks - continues to work with !terraform.output and !store functions

Files Changed

Core Implementation

• internal/terraform_backend/terraform_backend_gcs.go - GCS backend implementation
• internal/terraform_backend/terraform_backend_gcs_test.go - Comprehensive test suite
• internal/terraform_backend/terraform_backend_registry.go - Register GCS backend
• internal/terraform_backend/terraform_backend_utils.go - Updated error messages

Unified Authentication System

• internal/gcp/auth.go - New unified Google Cloud authentication (created)
• internal/gcp/auth_test.go - Authentication tests (created)
• pkg/store/google_secret_manager_store.go - Updated to use unified auth
• internal/gcp_utils/gcp_utils.go - Removed (replaced by unified auth)

Configuration & Documentation

• internal/exec/terraform_generate_backend.go - GCS backend validation
• website/docs/core-concepts/stacks/yaml-functions/terraform.state.mdx - Updated documentation
• errors/errors.go - Added GCS-specific error types
• go.mod - Added GCS storage dependency

Migration Guide

For users currently using !terraform.output or !store with GCS-stored state:

Before (slower)

# Using !terraform.output (requires Terraform execution)
vpc_id: !terraform.output vpc dev-us-east-1 vpc_id

# Using !store (requires separate state management)  
vpc_id: !store google-secret-manager dev/vpc/vpc_id

After (faster)

# Using !terraform.state (direct GCS state access)
vpc_id: !terraform.state vpc dev-us-east-1 vpc_id

Simply update your backend configuration to use gcs and replace function calls - no other changes needed!

Summary by CodeRabbit

• New Features

  • GCS-backed Terraform state support and unified Google Cloud authentication integration.

• Bug Fixes

  • Stricter backend config validation with clearer error responses and updated supported-backends messaging.

• Tests

  • Comprehensive unit tests added for GCS backend behavior and GCP authentication handling.

This PR addresses multiple authentication issues when using Atmos in containerized environments with mounted credential files:

1. Auth Pre-Hook Error Propagation - Terraform execution now properly aborts when authentication fails (e.g., Ctrl+C during SSO)
2. AWS Credential Loading Strategy - New LoadAtmosManagedAWSConfig() function provides proper isolation while preserving Atmos-managed profile selection
3. Noop Keyring Validation - Container auth now properly isolated from external environment variables
4. Whoami with Noop Keyring - atmos auth whoami now works in containerized environments
5. Test Coverage - Added test to verify auth errors properly abort execution

Changes

1. Auth Pre-Hook Error Propagation (internal/exec/terraform.go:236)

• Problem: Errors from auth pre-hook were logged but not returned, causing terraform execution to continue even when authentication failed (e.g., user presses Ctrl+C during SSO)
• Fix: Added return err after logging auth pre-hook errors
• Impact: Terraform commands now properly abort on auth failures

2. AWS Credential Loading Strategy (pkg/auth/cloud/aws/env.go)

• Problem: SDK's default config loading allowed IMDS access and was affected by external AWS_PROFILE, causing conflicts in containers
• Solution: Created LoadAtmosManagedAWSConfig() function that:
  • Clears credential env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  • Preserves profile/path vars (AWS_PROFILE, AWS_SHARED_CREDENTIALS_FILE, AWS_CONFIG_FILE)
  • Allows SDK to load from Atmos-managed credential files
• Impact: Proper isolation while still using Atmos-managed profiles

3. Noop Keyring Credential Validation (pkg/auth/credentials/keyring_noop.go)

• Problem: Used unrestricted config.LoadDefaultConfig() which allowed IMDS access and was affected by external AWS_PROFILE
• Fix: Changed to use LoadAtmosManagedAWSConfig()
• Impact: Container auth now properly isolated from external env vars

4. Whoami with Noop Keyring (pkg/auth/manager.go)

• Problem: Whoami() expected credentials from keyring, but noop keyring returns ErrCredentialsNotFound by design
• Fix: Added check for ErrCredentialsNotFound and fallback to buildWhoamiInfoFromEnvironment()
• Impact: atmos auth whoami now works in containerized environments

5. Test Coverage (internal/exec/terraform_test.go)

• Added TestExecuteTerraform_AuthPreHookErrorPropagation to verify auth errors properly abort execution
• Test validates that terraform doesn't continue on auth failure
• Updated test fixture to include required name_pattern configuration

Technical Details

The key insight is that Atmos sets AWS_PROFILE=identity-name (in pkg/auth/cloud/aws/setup.go:59) but the previous isolation approach cleared ALL AWS env vars including AWS_PROFILE. This caused the SDK to look for a non-existent [default] section.

The new LoadAtmosManagedAWSConfig preserves AWS_PROFILE while still preventing external credential conflicts.

...

Fixes stack configuration requirement for 6 commands that don't actually operate on stack manifests. These commands were incorrectly requiring stacks.base_path and stacks.included_paths to be configured, causing errors like:

Error: failed to initialize atmos config
stack base path must be provided in 'stacks.base_path' config or ATMOS_STACKS_BASE_PATH' ENV variable

What

Updated 6 commands to use processStacks=false in InitCliConfig:

Auth Commands (Commit 1)

• atmos auth env - Export cloud credentials as environment variables
• atmos auth exec - Execute commands with cloud credentials
• atmos auth shell - Launch authenticated shell

List/Docs Commands (Commit 2)

• atmos list workflows - List workflows from workflows/ directory
• atmos list vendor - List vendor configurations from component.yaml files
• atmos docs <component> - Display component README files

Why

These commands only need:

• Auth configuration from atmos.yaml
• Component base paths (terraform, helmfile, etc.)
• Workflow or vendor configurations

They do NOT need:

• Stack manifests to exist
• stacks.base_path to be configured
• stacks.included_paths to be configured

This makes Atmos more flexible for use cases like:

• CI/CD pipelines that only need auth or vendor management
• Development environments without full stack setup
• Documentation browsing without infrastructure configs
• Workflow management separate from stack operations

Technical Details

Changes Made

1. InitCliConfig parameter: Changed processStacks from true to false

   • Prevents validation requiring stacks.base_path and stacks.included_paths
   • Skips processing of stack manifest files

2. checkAtmosConfig option (for list vendor only): Added WithStackValidation(false)

   • Prevents checking if stacks directory exists
   • Required because list vendor calls checkAtmosConfig() with additional validation

Files Changed

• cmd/auth_env.go
• cmd/auth_exec.go
• cmd/auth_shell.go
• cmd/list_workflows.go
• cmd/list_vendor.go
• cmd/docs.go

Commands That Still Require Stacks (Unchanged)

These were NOT modified because they genuinely need stack manifests:

• atmos list stacks
• atmos list components
• atmos list settings
• atmos list values
• atmos list metadata

Testing

✅ All existing tests pass
✅ Linter passes with 0 issues
✅ Pre-commit hooks pass
✅ Manual testing confirms commands work without stack directories
✅ No regressions in existing functionality

References

Addresses user issue where atmos auth exec -- aws sts get-caller-identity failed with stack configuration error.

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

Summary by CodeRabbit

• New Features

  • Auth and utility commands (auth env, auth exec, auth shell, list workflows, list vendor, docs) now run without requiring stack configuration, enabling use in CI/CD, vendor management, and documentation workflows.

• Documentation

  • Added a blog post describing the change, usage examples, migration tips, and CI/CD benefits.

why

• Go releaser need more disk space

Summary by CodeRabbit

• Chores
  • Updated GitHub Actions runner specifications across feature release, nightly build, and test workflows to standardize build infrastructure configuration.

why

• Default runners have out of space

Summary by CodeRabbit

• Chores
  • Updated nightly release workflow to change how runner selection is provided: the workflow now accepts a JSON-like array of runner specifications, improving and broadening which runner(s) can be targeted for nightly builds.

why

• Terraform state operations need proper AWS credentials when accessing S3 backends
• Multi-identity scenarios require passing auth context through the call chain
• Reduces function parameter count from 6 to 2 (using PostAuthenticateParams struct)
• Simplifies nested conditional logic for better maintainability

references

• Part of multi-identity authentication context work
• Follows established authentication context patterns
• Related to docs/prd/auth-context-multi-identity.md

Summary by CodeRabbit

• New Features

  • Centralized per-command AuthContext enabling multiple concurrent identities (AWS, GitHub, Azure, etc.) and making in-process SDK and Terraform calls use Atmos-managed credentials.
  • Console session duration configurable via provider console.session_duration with CLI flag override.

• Bug Fixes

  • More reliable in-process authentication for SDK and Terraform state reads.

• Documentation

  • Added design doc, blog post, and CLI docs describing AuthContext and session-duration behavior.

• Tests

  • Expanded tests for auth flows, AWS config loading, and YAML/Terraform tag auth propagation.

why

• After commit 9fd7d15 (PR #1543), workflow shell commands lost access to PATH environment variable
• Users reported workflows that worked in v1.189.0 failed in v1.195.0 with commands like env, ls, grep not found
• This is a critical regression affecting any workflow using external executables
• Original fix conditionally replaced environment, which was inconsistent with executeCustomCommand behavior

Root Cause

The bug occurred in ExecuteShell() function in internal/exec/shell_utils.go:

1. Workflow commands call ExecuteShell with empty env slice: []string{}
2. ExecuteShell appends ATMOS_SHLVL to the slice: []string{"ATMOS_SHLVL=1"}
3. ShellRunner receives a non-empty env, so it doesn't fall back to os.Environ()
4. Shell command runs with ONLY ATMOS_SHLVL set, losing PATH and all other environment variables

Solution

Refactored ExecuteShell() to always merge custom env vars with parent environment:

// Always start with parent environment
mergedEnv := os.Environ()

// Merge custom env vars (overriding duplicates)
for _, envVar := range env {
    mergedEnv = u.UpdateEnvVar(mergedEnv, key, value)
}

// Add ATMOS_SHLVL
mergedEnv = append(mergedEnv, fmt.Sprintf("ATMOS_SHLVL=%d", newShellLevel))

This ensures:

• ✅ Empty env (workflows): Full parent environment including PATH
• ✅ Custom env (commands): Custom vars override parent, but PATH is preserved
• ✅ Consistent behavior: Matches executeCustomCommand pattern (line 393 in cmd_utils.go)

Testing

Unit Tests (internal/exec/shell_utils_test.go):

• TestExecuteShell/empty_env_should_inherit_PATH_from_parent_process - Verifies env command works
• TestExecuteShell/empty_env_should_inherit_PATH_for_common_commands - Tests ls, env, pwd, echo
• TestExecuteShell/custom_env_vars_override_parent_env - Verifies custom vars properly override parent

Integration Test (tests/test-cases/workflows.yaml):

• atmos workflow shell command with PATH - Full end-to-end workflow test using env | grep PATH

All tests pass, including existing workflow tests.

references

• Closes #1718
• Resolves DEV-3725
• Regression introduced in: 9fd7d15 (PR #1543)

Summary by CodeRabbit

• Bug Fixes

  • Shell commands now correctly inherit environment variables (including PATH) from the parent process, with custom env vars properly overriding parent values.

• Tests

  • Added tests covering environment inheritance for commands that require PATH, shell builtins, and custom env var overrides.

• Workflows / Snapshots

  • Added a workflow demonstrating PATH-dependent shell commands and updated related test snapshots and test cases.