probes: endpoint - #2696

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/routes/health/index.test.ts, line 74:

<comment>The exact `pressure` assertions depend on the CI host's real memory. `machineVitals` (libs/machine.ts) pushes a `memory at X% of the machine` entry whenever `Deno.systemMemoryInfo()` reports ≥90% used — a value this test cannot control. On a shared/oversubscribed runner, `toEqual([])` (line 48) and this exact `toEqual([...])` will fail spuriously whenever host memory crosses 90%, even though every dependency is up and saturation is exactly 120/128. Assert only what the test controls: use `toContain` for the in-flight pressure reason instead of an exact-array equality.</comment>

<file context>
@@ -0,0 +1,124 @@
+    });
+  });
+
+  test("a saturated machine is degraded even with every dependency up", async () => {
+    const res = await route([ok("database", true)], { inFlight: 120 }).request(
+      "/health",
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/index.ts, line 238:

<comment>When a dependency probe fails, this public `/health` endpoint exposes the truncated underlying exception in `checks[].error`, which can disclose internal hosts or connection details. Return generic failure text for unauthenticated callers, or protect the diagnostic response separately from the public readiness status.</comment>

<file context>
@@ -224,6 +229,19 @@ app.get("/ping", (c) => {
+ */
+app.route(
+  "/",
+  createHealthRoute({
+    probes: probesFromEnv(),
+    inFlight: () => concurrencyGuard.inFlight(),
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/libs/health.ts, line 76:

<comment>When a non-abortable probe exceeds its deadline, `Promise.race` returns `down` but leaves `probe.run` running; the configured `db.run` ignores `signal`. Repeated health rounds during a database outage accumulate requests until client timeouts, so make this probe cancellable or back off timed-out work.</comment>

<file context>
@@ -0,0 +1,113 @@
+  }, probe.timeoutMs);
+
+  try {
+    await Promise.race([probe.run(controller.signal), expired]);
+    return {
+      ...base,
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/libs/health.ts, line 87:

<comment>Because `/health` is unauthenticated, `error: describe(error)` exposes the first 200 characters of dependency errors, not a redacted form. Credentials or internal connection strings can therefore be returned when a dependency includes them; expose only fixed safe error codes or redact sensitive values.</comment>

<file context>
@@ -0,0 +1,113 @@
+      ...base,
+      status: "down",
+      latencyMs: Math.round(performance.now() - startedAt),
+      error: describe(error),
+    };
+  } finally {
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/env.ts, line 15:

<comment>The comment says these vars are "reported by `/ping`", but `/ping` only returns `region` and `requestId`. `FLY_MACHINE_ID`/`FLY_APP_NAME`/`FLY_MACHINE_VERSION` are consumed in `libs/machine.ts` and surfaced through `/health`, not `/ping`. Fix the comment so it doesn't mislead debugging.</comment>

<file context>
@@ -12,6 +12,10 @@ export const env = createEnv({
     UPSTASH_REDIS_REST_URL: z.string().min(1),
     UPSTASH_REDIS_REST_TOKEN: z.string().min(1),
     FLY_REGION: z.enum(monitorRegions),
+    // Injected by Fly at boot, reported by `/ping`. Absent locally and in tests.
+    FLY_MACHINE_ID: z.string().optional(),
+    FLY_APP_NAME: z.string().optional(),
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/libs/machine.ts, line 127:

<comment>When the wall clock is adjusted, `uptimeSeconds` becomes incorrect even though the process keeps running. Measure uptime with a monotonic source such as `Deno.uptime()` or `performance.now()`.</comment>

<file context>
@@ -0,0 +1,151 @@
+    machine: {
+      ...identity,
+      host: hostname,
+      uptimeSeconds: Math.round((now - bootedAt) / 1000),
+    },
+    cpu: {
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/libs/machine.ts, line 52:

<comment>When the system clock moves backward, this condition remains true until wall time catches up, so `/health` can serve stale memory and load metrics. Use a monotonic clock, or require `now >= cached.at` before reusing the sample.</comment>

<file context>
@@ -0,0 +1,151 @@
+let cached: { at: number; sample: HostSample } | null = null;
+
+function sampleHost(now: number): HostSample {
+  if (cached && now - cached.at < SAMPLE_TTL_MS) return cached.sample;
+  const sample: HostSample = {
+    process: Deno.memoryUsage(),
</file context>

Check if this issue is valid — if so, understand the root cause and fix it. At apps/server/src/libs/health.test.ts, line 67:

<comment>The comment claims this test proves probes run concurrently, but the assertion cannot detect a regression to serial execution. The mock probes here both finish almost instantly (redis times out at ~20ms, database resolves immediately), so the report latency is ~20ms whether runProbes uses Promise.all or loops serially — both satisfy `toBeLessThan(200)`. To actually guard the concurrency property, give the resolving probe a measurable delay (e.g. 30ms) and assert latency is below the serial sum rather than a loose 200ms bound.</comment>

<file context>
@@ -0,0 +1,156 @@
+    expect(report.status).toBe("degraded");
+    expect(report.checks[0].error).toBe("timed out after 20ms");
+    // Concurrent: the whole report waits for the slowest probe, not their sum.
+    expect(report.latencyMs).toBeLessThan(200);
+  });
+
</file context>