environments: build secrets (E3-05) - #28

#28 landed ManagedBuilder.supports_build_secrets generically -- daytona.py
already sets it False, and _shared_findings() (called alongside
_own_findings() in validate()) now refuses spec.buildSecrets on its own.
This branch's own hand-rolled refusal in _own_findings would have doubled
the finding; removed, with a one-line note pointing at the flag instead.

763 passed, 4 skipped (full -k environment run); ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Same consolidation as feat/daytona-builder: supports_build_secrets =
False already refuses spec.buildSecrets generically via
ManagedBuilder._own_findings, called alongside this class's own
_own_findings in validate(). This branch's hand-rolled refusal would have
doubled the finding; removed.

752 passed, 4 skipped (full -k environment run); ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…w that resolve_build_secret exists

Unlike E2B/Daytona, Modal never set supports_build_secrets = False --
this builder's own refusal was always the real, deliberate one (Modal has
a genuine per-step secrets= mechanism; only the wiring was missing). #28
landed resolve_build_secret on main, so "not yet merged" is no longer
true -- updated every refusal message to say what actually remains
(nothing here calls it yet), and pointed at the real module in the
docstring.

test_environment_managed_builders.py's test_datalayer_and_modal_accept_a_build_secret
was written against the pre-E2-05 stub, where Modal (no _own_findings
override yet) accepted anything -- split into
test_datalayer_accepts_a_build_secret (still true) and
test_modal_refuses_a_build_secret_too_but_not_for_lack_of_a_mechanism
(the real builder's own, different reason).

Wiring resolve_build_secret into an actual Modal secrets= collection,
with its own creation/cleanup lifecycle, is real follow-up work, not
attempted here.

764 passed, 4 skipped (full -k environment run); ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* environments: the Daytona builder — build, inspect, exists (E2-04)

Implements adapters/daytona.py's build/inspect/exists against the
Daytona SDK's declarative Image and snapshot API (section 11.3):

- The declarative image starts from the resolved Datalayer base by
  digest, pulled through a registry entry made and torn down around
  this build alone (D-17, D-18) — Daytona's own ECR option takes only
  a standing role, not a per-build credential.
- USER root brackets the apt install and uv pip sync; no synthetic
  account is created, because Daytona honours the base's own USER,
  HOME and WORKDIR (confirmed live), unlike E2B (E2-03).
- Neither the doctor nor the wheelhouse is copied in: the Datalayer
  base already bakes both (E1-05), confirmed live — a directory COPY
  merges into what the base already has rather than replacing it, so
  only the lock is genuinely per-build.
- An explicit tini -- sleep infinity entrypoint replaces Daytona's own
  unset default, which is alive but not a real PID 1.
- Resources come from a local CPU size-class table (this package
  publishes to PyPI; the canonical one, datalayer_common.size_classes,
  does not); a GPU size class is left buildable at validate() (an
  existing test already pins that answer) and refused at build() time
  instead, naming E2-17, since bases.py cannot resolve a CUDA digest
  yet.
- provider_account (E2-01) is populated from the credential; get/
  exists map provider failures to DL_ENV_PROVIDER_ERROR.

Two live drills (registered a private registry against the real ECR
base, built a snapshot, launched a sandbox, ran commands, deleted
everything) found: the registry entry must be deleted by id, not the
name it was given; and that a wheelhouse copied in duplicates rather
than replaces what the base already has.

36 new tests (test_environment_daytona_builder.py) against daytona/
daytona_api_client doubles; test_environment_managed_builders.py's
build-refuses-by-name tests narrowed now that Daytona is built.

resolve, delete and smoke_test still refuse via the inherited
ManagedBuilder — a live drill with a real resolved lock and the
formal core tier is still owed before E2-04 ticks (see plans/ENV.md).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* environments: address the review — JWT auth, no redundant uv, build secrets

Three real gaps Copilot's review of PR #30 found, all fixed:

- _client() only read DAYTONA_API_KEY, so a JWT-authenticated owner
  (accounts.py's own CREDENTIAL_VARIABLES lists both forms) fell
  through to the worker's ambient credentials instead of building in
  their own organization. Now reads DAYTONA_JWT_TOKEN +
  DAYTONA_ORGANIZATION_ID as a fallback.
- The chain reinstalled uv with pip, even though the approved base
  already bakes it (E1-05; resolve.py's own bootstrap_uv docstring
  says so) - an un-hashed network fetch outside the resolved lock for
  no reason. Removed, matching the Datalayer builder's own
  dockerfile(), which installs uv only for the image source (not
  built for this variant yet).
- spec.buildSecrets was neither consumed nor refused: a supported
  environment naming one would silently build without it. Refused in
  _own_findings now, the same reasoning as E2B's.

4 new tests; 39 total in test_environment_daytona_builder.py, all
passing. ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* environments: the build-time doctor check cannot pass on Daytona — remove it

Found live with a real, hash-verified, 310-package resolved lock (the
full protected constraint set, not the trivial locks earlier drills
used): `datalayer-sandbox doctor --json` failed its `init` row
deterministically, twice, with "pid1": "python3" and an unreaped
orphan. A Daytona build-time RUN step does not execute as the final
snapshot's own PID 1 — it runs inside Daytona's own build agent's exec,
whose PID 1 is that agent's own python3 process, not tini or whatever
the finished snapshot actually starts as. Every other doctor row (uid,
gid, workdir, the kernel stack, ...) passed correctly in that same
build; only `init` cannot be truthfully answered from inside a build
step, for any provider whose build runs one command at a time this
way, not only Daytona's.

The doctor --json build step is removed. A real answer to "does this
identity actually start correctly" needs a launched sandbox, which is
a smoke test's job (E1-14), not this builder's — matching what the
Datalayer and E2B builders already do differently for their own
reasons, and what this same builder already decided for the entrypoint
question.

Confirmed live, end to end, with the fix: the same 310-package lock
now builds to Active, launches through the real Sandbox.create path,
and passes all 9 checks of the formal core tier (doctor, identity,
python, kernel, imports, filesystem, restart, shutdown, secrets) —
E2-04's own "Done when" criterion.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* docs/daytona: document the Environments builder (E2-13, partial)

What a Daytona-built artifact is (a snapshot, from the approved base),
bring-your-own-account (the owner's own organization via DAYTONA_API_KEY
or the JWT+org-id pair, never this package's ambient credentials), what's
specific to Daytona's own artifact (resources and region, since a
size-class or target change is a second, distinct artifact), and the
build-time-doctor-vs-live-launch story (the one variant where they fully
agree: not run at build time, on purpose, and a real launch passes the
core tier in full).

Partial slice of E2-13: the UI docs, clouder docs and per-provider make
targets it also names are out of scope here, most of them blocked on
infrastructure and product API work this repo has no reach into.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* merge main (E3-05, #28): drop the now-redundant buildSecrets refusal

#28 landed ManagedBuilder.supports_build_secrets generically -- daytona.py
already sets it False, and _shared_findings() (called alongside
_own_findings() in validate()) now refuses spec.buildSecrets on its own.
This branch's own hand-rolled refusal in _own_findings would have doubled
the finding; removed, with a one-line note pointing at the flag instead.

763 passed, 4 skipped (full -k environment run); ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>

* environments: the E2B builder — build, inspect, exists (E2-03)

Implements the build half of section 11.2: a template from the resolved
lock, starting from E2B's own code-interpreter-v1 rather than the Datalayer
ECR base — that base's own patched code-interpreter server (E0-04: the
FastAPI/Jupyter service on 49999/8888) is proprietary and not published
anywhere this package could vendor it from, so the chain starts from
E2B's own template instead and reconciles it to the lock the same way
E1-07 already reconciles a base's own package versions. adapters/e2b.py's
own module docstring has the full reasoning.

- build(): copies the doctor (built fresh via doctor/build.py), the
  wheelhouse and the lock in, uv pip sync --require-hashes on top of
  code-interpreter-v1's own packages, files_step()'s baked files and
  postInstall run under the datalayer account set_user creates, and the
  contract's own doctor check at build time. The artifact is
  <team>/<name>:<build_id> (E0-04's own verified format); the template id
  travels in ArtifactReference.mutable_alias for inspect/exists to use
  without a second lookup.
- inspect()/exists(): read the template's tags for the one matching this
  build's id.
- resolve()/delete()/smoke_test() still refuse via the inherited
  ManagedBuilder — smoke_test deliberately so, since nothing on the real
  build workflow calls it (durable's own _SmokeTest seam is gated on
  E1-14, with a different signature); see the module docstring.

Three real findings from live builds against a throwaway template
(code_sandboxes/environments/adapters/e2b.py's own comments carry the
full account): copy()'s source must be relative to a context directory;
code-interpreter-v1's persistent default user means root steps need an
explicit user="root"; and set_user("datalayer") must come after a real
useradd, never before one exists, or the account can't exec anything at
all. What remains, live-verified but not yet passing: code-interpreter-v1
already holds uid 1000 or gid 100 under another account, so datalayer
lands on 1001:1001 instead of the contract's 1000:100, and the doctor
correctly refuses the build on it — E2-03 stays unticked in plans/ENV.md
until that numeric mismatch is resolved.

Tests: 17 new (test_environment_e2b_builder.py) with a Template/
TemplateBuilder double, plus 2 existing tests updated for the operations
E2B now actually implements. 715 passed on the full environments suite,
ruff and mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* environments: address the review — credential threading, apt, env order, more

Six real findings from this PR's own review, all fixed:

- The owner's credential (D-8) is now actually passed to the E2B SDK —
  Template.build/get_tags/exists all take api_key= from
  BuildCredential.provider_secrets when one is given, falling back to the
  SDK's own ambient E2B_API_KEY otherwise. Without this, a multi-owner
  worker would have built in whichever team the worker process itself was
  configured for, never the environment owner's.
- spec.packages.system.apt's locked pins are now installed (apt-get, as
  root, before uv pip sync) — this chain used to run uv pip sync alone and
  silently ship a spec's system dependencies missing.
- set_envs moved before any install step, matching the Datalayer builder's
  own reasoning (E1-07): a package that compiles against a library found
  through an env var behaves differently without it.
- A spec naming buildSecrets is now refused in _own_findings, the same
  way this variant already refuses a GPU class — E0-04's spike found no
  per-step secret mechanism for E2B, only a registry login.
- ArtifactReference.provider_account is now populated from the credential
  (E2-01) — previously always unset, so a launch could never verify the
  account an artifact was built in.
- get_tags/exists failures now map to DL_ENV_PROVIDER_ERROR instead of
  escaping as whatever the SDK happens to raise, matching the Datalayer
  registry adapter's own provider-call discipline.

The uid:gid gap (Copilot's first comment) is the one already documented in
this PR's own description and the module's "what remains" note — not new,
and not something this pass attempts again.

Tests: 11 new (env ordering, apt install, credential threading, provider
account, error mapping, build-secret refusal), 28 in this file total, 726
on the full environments suite. ruff, mypy and ruff-format clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* e2b: close the uid/gid, locale and post-build-collision gaps; find a deeper runtime one

Three live-confirmed bugs, fixed:

- The account created for the contract never actually landed on uid 1000,
  gid 100: code-interpreter-v1 already holds an account at uid 1000 (`user`)
  and a group at gid 100 (`users`) of its own, so `useradd -u 1000 -g 100`
  silently fell back to the next free numbers instead of erroring. Fixed by
  renaming whoever already holds uid 1000 (found live, via a build-free
  probe of code-interpreter-v1) rather than creating a second account,
  falling back to a plain useradd if a future base has no such account.

- The doctor's own locale check failed: code-interpreter-v1 sets no locale
  at all, unlike the Datalayer base (E1-05) Daytona and Modal both inherit
  it from. Fixed by setting LC_ALL/LANG=C.UTF-8 unconditionally, merged
  under the spec's own env so a spec can still override it.

- Once the identity fix let a build get far enough, it collided with E2B's
  own post-build "configuration script", which unconditionally recreates a
  default `user` account and failed outright ("group user exists") because
  the renamed account's own now-orphaned private group was still there.
  Fixed by deleting that leftover group (never the gid-100 group itself).

With all three fixed, a real build now passes `datalayer-sandbox doctor
--json` in full at build time (live-verified). A live launch through the
real launcher still fails the core tier's identity checks, though: a
fourth, deeper gap, fully root-caused and documented in the module
docstring rather than chased further here (same reasoning as
modal_sandbox.py's still-open setpriv gap, E2-05) — jupyter.service and
code-interpreter.service run as root regardless of set_user/set_workdir,
and E2B's own private main.py/jupyter_server_config.py hardcode
/home/user as the cwd, independent of which account runs the process.

29 tests, all passing; ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* docs/e2b: document the Environments builder, including the open identity gap (E2-13, partial)

What an E2B-built artifact is (a template, starting from code-interpreter-v1
rather than the Datalayer base, since E2B's own code-interpreter server is
proprietary and only ships there), bring-your-own-account (the owner's own
E2B team), and the honest, current state of the identity gap: the build
itself gets uid/gid/locale right, but a launched sandbox still runs as
root because the two systemd services that actually execute code have no
User= of their own and E2B's private server hardcodes /home/user,
independent of anything the build sets.

Partial slice of E2-13, same scope note as the Daytona docs commit.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* merge main (E3-05, #28): drop the now-redundant buildSecrets refusal

Same consolidation as feat/daytona-builder: supports_build_secrets =
False already refuses spec.buildSecrets generically via
ManagedBuilder._own_findings, called alongside this class's own
_own_findings in validate(). This branch's hand-rolled refusal would have
doubled the finding; removed.

752 passed, 4 skipped (full -k environment run); ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>

* environments: the Modal builder — build, inspect, exists (E2-05)

Implements adapters/modal.py's build/inspect/exists against Modal's
own image builder and Secret/App/Client APIs (section 11.4):

- The base is pulled through modal.Image.from_aws_ecr with a Secret
  made from this build's own credential (D-17, D-18) — IAM-shaped
  (AWS_ACCESS_KEY_ID/SECRET/REGION), not the docker-login pair Daytona
  and E2B take. Deleted afterward through the raw SecretDelete RPC,
  reached via synchronizer.wrap since Modal exposes no plain method
  for deleting an unnamed secret.
- No USER line is ever emitted: Modal silently ignores USER entirely
  (found live), so every build step already runs as root regardless.
  For the same reason, the contract's own doctor check is not run at
  build time — it would report the wrong identity; that is a live
  launch's job (E1-14), where the sandbox launcher re-asserts uid 1000
  via setpriv on every exec.
- Neither the doctor nor the wheelhouse is copied in: the Datalayer
  base already bakes both (E1-05), the same finding as Daytona's.
- MODAL_IMAGE_BUILDER_VERSION is pinned to 2025.06 every build: the
  2023.12 default fails compiling Modal's own runtime deps against
  Python 3.13 (found live, matching E0-04's own recorded failure).
- The entrypoint execs its arguments via a placeholder-free $0 $@
  form, not the more usual quoted positional-parameter idiom:
  entrypoint() does not escape an embedded quote when rendering the
  Dockerfile ENTRYPOINT array (found live: the quoted form produced a
  sandbox that shut down within seconds).
- A GPU size class is left buildable at validate() (matching an
  existing test) and refused at build() time instead, naming E2-17,
  the same reasoning as Daytona's.
- spec.buildSecrets is refused: resolve_build_secret (E3-05) is a
  separate, not-yet-merged branch, even though Modal's own per-step
  secrets mechanism could carry one once it lands.
- inspect/exists are honestly weaker than the other variants': found
  live, Image.from_id does not distinguish a deleted image (after
  experimental.image_delete) from one still there, across four
  different ids — documented plainly rather than implying a guarantee
  that is not kept.

Two live drills (a hand-rolled build against the real ECR base and
Modal workspace, then a full run through this exact Builder class)
found and fixed: add_local_file only records a path, read lazily
inside image.build() rather than eagerly, so the scratch directory
must outlive the whole build, not just the chain construction; and a
build failure's own log was silently dropped because the capture
buffer was only drained on the success path.

35 new tests (test_environment_modal_builder.py) against a modal SDK
double; test_environment_managed_builders.py's build-refuses-by-name
tests narrowed now that Modal is built.

resolve, delete and smoke_test still refuse via the inherited
ManagedBuilder.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* environments: address the review — real entrypoint script, error mapping, honesty

Five real gaps Copilot's review of PR #31 found, all fixed:

- The entrypoint's own rationale claimed code_sandboxes/modal_sandbox.py
  already re-asserts identity with setpriv on every exec. It does not:
  ModalSandbox execs directly, so an environment built here and
  launched today runs as root, not 1000:100. The doctor check is still
  not run at build time (it would report the wrong identity there
  too), but the module docstring now says so plainly instead of
  describing a launcher fix that has not landed.
- A build's own Secret could be created server-side by `hydrate()` and
  then still raise on this process's own observation of the call,
  leaving the credential behind with nothing local pointing at it.
  `_ecr_secret` now attempts the same best-effort delete before
  re-raising.
- The unquoted `exec $0 $@` entrypoint (from the previous review round,
  fixing a different bug) loses argument boundaries through word
  splitting and glob expansion — a `python -c "..."` source, above
  all. Replaced with a real script file baked in via add_local_file:
  `#!/bin/sh\nexec "$@"\n` is file content, not a Dockerfile-rendered
  argv string, so its own quotes need no escaping, and `entrypoint()`
  is given one bare path with nothing to mis-render. Confirmed live: a
  multi-word python -c argument now arrives as one argument.
- `_client` is called before build/inspect/exists's own try blocks, so
  Client.from_credentials/from_env failures escaped as raw Modal
  exceptions instead of the adapter's own taxonomy. Now mapped to
  DL_ENV_PROVIDER_ERROR.
- build_secret_ids/spec.build_secrets was only refused at validate()
  time; build() itself had no guard, so a caller handing it an
  already-resolved request could get a successful image with a
  required credential silently missing. build() now refuses it too,
  the same way the GPU-class guard already does.

7 new tests; 44 total in test_environment_modal_builder.py, all
passing. A third live drill (the real Builder class, end to end)
confirms the new entrypoint script keeps a sandbox alive and, exec'd
with an explicit setpriv the way a real launch is meant to, reports
the correct 1000:100 identity — the same drill's own hand-rolled exec,
not yet what modal_sandbox.py itself does. ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* environments: the entrypoint must stay alive with no command at all

Found live, 2026-09-13, building a new live matrix test (E2-11) that
runs this exact builder's own artifact through the real launcher for
the first time, rather than a hand-rolled Sandbox.create call:
code_sandboxes.modal_sandbox.ModalSandbox.start() creates the sandbox
with no command arguments at all, and execs into the running
container separately, afterward. exec "$@" with nothing to expand is
a no-op in sh, so the entrypoint script fell straight through to its
own end and the container exited before that first real exec ever
reached it -- every environment built by this adapter and launched
through the real ModalSandbox was silently dying immediately.

The entrypoint now execs `sleep infinity` when given no arguments, and
`"$@"` when it is given one -- correct either way a caller invokes it.
Confirmed live: the container now stays alive through a full core-tier
run (python, kernel, restart, shutdown and secrets checks all passed
against a real launched sandbox; only the already-documented identity
gap, root instead of 1000:100, still fails, which is what modal_sandbox.py's
own missing setpriv wrapper is expected to cause).

2 new tests in test_environment_modal_builder.py.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* modal_sandbox.py: put the contract's identity back for a contract artifact only

Section 11.4 item 6 asks for this: "every exec re-asserts the user" with a
setpriv-style wrapper, since Modal ignores the image's own USER. Nothing in
modal_sandbox.py did that until now.

`_start_driver` sets three env vars on the session driver's own exec() call
(DATALAYER_SANDBOX_CONTRACT_UID/GID/HOME) and a workdir, but only when
self._image_id is set -- that is, only when this ModalSandbox was launched
from a built Environments artifact, whose image adapters/modal.py's own
build recipe always chowns to 1000:100 regardless of the ignored USER line.
The driver itself (_DRIVER_SOURCE, a script this package owns end to end,
unlike Modal's own execution engine) reads those vars and calls
os.setgid()/os.setuid() before entering its request loop, only if it is
actually running as root and the target ids are real in this image --
anything else leaves it exactly as it started rather than crash a session
that could otherwise still run.

A plain ModalSandbox (debian_slim, or anyone else's image, no image_id at
all) never has these set and drops nothing -- general Modal sandbox usage,
well beyond Environments, is unaffected on purpose (the risk this deferred
the fix for last time: an unconditional drop against an account that
image may not have).

modal.Sandbox.exec() has no user= of its own (confirmed against the
installed SDK's signature), which is why this lives in the driver rather
than as an exec-time kwarg.

Live-verified against a real build and a real launch: doctor and identity
(checks 1, 2) now pass, where they used to report uid 0. Checks 5 and 6
(imports, filesystem) still fail -- confirmed, with this fix in place, to
be a separate, pre-existing issue unrelated to identity: a plain,
non-Environments ModalSandbox runs 8 sequential snippets with no trouble
at all, so this is specific to a contract-built artifact's image under
repeated exec, not the privilege drop. Not chased further here; recorded
for whoever picks up E2-05's remaining box next.

stop() also gets the orphan-on-partial-start fix from code-sandboxes#32
(guarded on self._sandbox, not _started), so this branch does not regress
it once the two merge in either order.

16 tests, all passing; ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* docs/modal: document the Environments builder, including the identity fix and the open imports/filesystem gap (E2-13, partial)

What a Modal-built artifact is (an image id, from the approved base),
bring-your-own-account (the owner's own workspace via
MODAL_TOKEN_ID/SECRET), and the identity story in full: Modal ignores the
image's own USER, but the session driver now drops itself to the
contract's 1000:100 for an Environment-built image specifically, gated on
image_id so general Modal sandbox usage is unaffected -- confirmed live.
The two checks that still fail (imports, filesystem) for a separate,
not-yet-root-caused reason are documented honestly rather than implied
fixed.

Partial slice of E2-13, same scope note as the Daytona/E2B docs commits.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* merge main (E3-05, #28): update the build-secrets refusal wording, now that resolve_build_secret exists

Unlike E2B/Daytona, Modal never set supports_build_secrets = False --
this builder's own refusal was always the real, deliberate one (Modal has
a genuine per-step secrets= mechanism; only the wiring was missing). #28
landed resolve_build_secret on main, so "not yet merged" is no longer
true -- updated every refusal message to say what actually remains
(nothing here calls it yet), and pointed at the real module in the
docstring.

test_environment_managed_builders.py's test_datalayer_and_modal_accept_a_build_secret
was written against the pre-E2-05 stub, where Modal (no _own_findings
override yet) accepted anything -- split into
test_datalayer_accepts_a_build_secret (still true) and
test_modal_refuses_a_build_secret_too_but_not_for_lack_of_a_mechanism
(the real builder's own, different reason).

Wiring resolve_build_secret into an actual Modal secrets= collection,
with its own creation/cleanup lifecycle, is real follow-up work, not
attempted here.

764 passed, 4 skipped (full -k environment run); ruff/mypy clean.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>