Reuse production dependencies when application wheels change - #73092

• depends-on: apache/airflow#73090
• diff for early review: ci/image-reuse/01-frozen-dependencies...ci/image-reuse/02-prod-dependency-layers
• next: apache/airflow#73093

Part 2/5 of the CI image reuse series.

Why

Avoid reinstalling production dependencies when only application code changes.

How

Separate dependency metadata from application wheels, reuse the dependency layer, and install current wheels with normal dependency checks.

Series impact (audit draft)

Sept 6–12, 2026 UTC PR census and CI timing:

The purpose is to avoid rebuilding the CI image when a PR doesn't change package dependencies — currently, even a PR that only touches the API server or the UI still triggers a full CI image rebuild on every push — and real hit rate and end-to-end PR latency remain unmeasured pending a pilot. None of this changes this PR's own diff.

• Happy path: a cache hit cuts image-prep latency by ~71% (19.74 → 5.77 min for the dominant Python 3.10 CI case). A miss only costs about 30 extra seconds, so even a low cache-hit rate (as little as ~3.5%) already makes this a net time saver overall.
• Smart fallback: the consumer probes transfer speed over the first 30 seconds of a selected image's download; if that regresses below the reuse floor (added in ci/image-reuse/03-artifact-selection), it falls back to a regular image build — today's existing behavior — instead of waiting out a degraded transfer.
• Daily image publish workflow: a centralized periodic workflow publishes CI and production images, keyed by a fingerprint of build inputs (dependency lockfiles, base-image digest, build args), as GitHub Actions artifacts for PRs to reuse.
• Retention: a published artifact is only selected if produced within the last 48 hours, and is retained for 7 days.

Was generative AI tooling used to co-author this PR?

• Yes, with help of Codex (GPT-6) following the guidelines.

Generated-by: Codex (GPT-6)