E2-11: nightly live matrix for the managed Environment builders - #32

Adds a nightly live matrix that builds, launches, and validates managed Environment artifacts across Daytona, E2B, and Modal.

Changes:

• Adds provider live tests and core-tier validation.
• Adds make environments-live.
• Adds scheduled/manual workflow with failure issue reporting.

Review findings: eight moderate issues remain, covering workflow permissions and timeout, warning filters, build ID collisions, partial-start cleanup, exception preservation, provider-specific base resolution, and Daytona build-context cleanup.

Suppressed comments (13)

.github/workflows/environments-live.yml:34

• The three tests run sequentially, but the provider builders allow up to 30 minutes for Daytona, 30 minutes for Modal, and 20 minutes for E2B, before resolution and launch time. A valid slow run can hit this 45-minute hard limit while a provider is still active, skipping cleanup and preventing later providers from running. Increase the timeout or split this into per-provider jobs.

    timeout-minutes: 45

tests/test_environment_live_matrix.py:55

• Pytest is configured with filterwarnings = ["error", ...] (pyproject.toml:95-106), and the other live suites explicitly ignore provider SDK deprecations. This module imports Modal/Daytona/E2B SDKs without those marks, so a client warning can fail a row before any sandbox check; add the live warning filters here.

pytestmark = [pytest.mark.live]

tests/test_environment_live_matrix.py:181

• Because this module-scoped fixture is set up before the test body, _resolved_lock() runs before _skip_unless_available(). When live mode or provider/AWS credentials are missing, the workflow still performs uv/PyPI resolution (and can fail) instead of skipping immediately as documented. Gate the fixture before resolving, or resolve the lock only after the per-provider availability check.

    lock_text, resolved_bases = _resolved_lock()

tests/test_environment_live_matrix.py:155

• BuildRequest.lock_digest is documented as the digest of lock_text, but this passes an all-zero placeholder rather than resolve_environment()'s returned digest. Any builder metadata or attestation that uses this field will advertise a hash that cannot verify the lock installed by this test; carry the resolver's digest through the fixture.

        lock_digest="sha256:" + "00" * 32,

tests/test_environment_live_matrix.py:226

• This xfail covers the entire test body, so provider outages, bad credentials, SDK/import failures, and build/launch regressions are all reported as expected failures—not just the documented root-identity gap. The nightly workflow will therefore miss genuine Modal failures; verify the expected check IDs first and only xfail that specific outcome.

    @pytest.mark.xfail(
        reason="modal_sandbox.py has no setpriv wrapper yet: execs run as root, not "
        "the contract's 1000:100 — checks 1, 2, 5 and 6 all fail on it",
        strict=False,

tests/test_environment_live_matrix.py:264

• This blanket marker also turns get_builder("e2b")/build() failures such as the current DL_ENV_CAPABILITY_UNSUPPORTED "has not landed" state into xfails, contrary to the module's stated expectation that an unmerged builder fails outright. It likewise hides unrelated provider regressions; restrict xfail to the verified uid/gid failure and let other exceptions fail the job.

    @pytest.mark.xfail(
        reason="E2-03's own documented gap: code-interpreter-v1 lands datalayer "
        "on uid 1001, gid 1001, not the contract's 1000:100",
        strict=False,

tests/test_environment_live_matrix.py:283

• This finally is a no-op, so every successful E2B run leaves its remote template behind; if build() fails before artifact is assigned, this block is not reached at all. The nightly job will accumulate provider artifacts and costs, contradicting the module's cleanup guarantee. Add provider-side deletion (for example through the provider API) or do not schedule this row until cleanup is available.

        finally:
            pass  # E2B's SDK exposes no template delete call (found live, E2-03).

tests/test_environment_live_matrix.py:244

• ModalSandbox.start() creates the remote sandbox before _started is set, and stop() is a no-op when startup fails before that flag is set. Thus an exception during driver/context setup can leave a Modal sandbox running because this cleanup is only entered after start() returns. Make partial-start cleanup explicit in the provider lifecycle.

            sandbox.start()

tests/test_environment_live_matrix.py:277

• E2BSandbox.start() creates the remote sandbox before setting _started, while stop() returns when _started is false. If context setup fails after creation, this cleanup is skipped and the live matrix leaks the sandbox. Make partial-start cleanup explicit in the provider lifecycle rather than relying on start() to complete.

            sandbox.start()

tests/test_environment_live_matrix.py:252

• Deleting only artifact.provider_artifact_id is not sufficient for Modal: its chained image build leaves intermediate layer images that the final image deletion does not remove. Each nightly build will therefore accumulate Modal artifacts despite the module's cleanup guarantee; enumerate the intermediate ids through the provider cleanup/reconciliation path as well.

            modal.experimental.image_delete(
                artifact.provider_artifact_id, client=modal.Client.from_env()
            )

tests/test_environment_live_matrix.py:209

• The artifact delete in finally is unguarded, so a provider cleanup error replaces the original build/core-tier exception and the nightly issue loses the actual check failure. Preserve the primary exception while recording cleanup failure separately.

            import daytona

            daytona.Daytona().snapshot.delete(artifact.provider_artifact_id)

tests/test_environment_live_matrix.py:125

• The lock is resolved with variants=['daytona'], then the Daytona base is reused for E2B and Modal. resolve_environment returns a per-variant base map, so this matrix will build the wrong base or fail to exercise provider-specific base pins as soon as those variants diverge. Resolve all three variants and select resolved_bases[variant] per test.

    outcome = resolve_environment(
        spec=_ENVIRONMENT_SPEC, variants=["daytona"], runner=LocalResolveRunner()
    )
    return outcome["content"], outcome["resolved_bases"]

tests/test_environment_live_matrix.py:209

• snapshot.delete(...) removes the snapshot id, but Daytona keeps the build context in provider object storage and this path has no cleanup for it. The nightly test therefore accumulates remote build data even when snapshot deletion succeeds, contradicting the claim that everything it creates is deleted. Add provider-specific context cleanup or document and bound the retained resource.

            import daytona

            daytona.Daytona().snapshot.delete(artifact.provider_artifact_id)

• Files reviewed: 3/3 changed files
• Comments generated: 3
• Review effort level: Lite