Thanks to visit codestin.com
Credit goes to github.com

Skip to content

iOS runner smoke: alert-observation XCTests are red or flaky on main #2546

Description

@thymikee

Purpose

Smoke Tests in the iOS workflow is not deterministic on main. Two alert-observation XCTests fail on the runner host without any Apple source changing, so every PR that reaches that step has to be re-run before a reviewer can trust it.

Observed on main

  • main@973b74cc14 (push run 34741577232, job 103682270067, step Run targeted iOS runner XCTest regressions), RunnerTests+AlertObservationTests.swift:72:

    XCTAssertEqual failed: ("false") is not equal to ("true")
    - Optional(… code: "ALERT_DEADLINE_EXCEEDED", message: "alert dismiss exhausted its deadline")
    

    Failing test: testAlertDismissDoesNotActivateAReplacementWithTheSameTitle.

  • PR docs: fix phantom specifiers, the duplicate ADR 0019, and the Node floor #2533 head a16821b661 (run 34742468875 attempt 1, docs-only diff), same file at :75 and :80:

    XCTAssertEqual failed: ("First actions: 0; replacement actions: 0") is not equal to ("First actions: 1; replacement actions: 0")
    

    Failing test: testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt. Attempt 2 of the identical run and commit passed at 06:46 UTC, so the second case is a flake rather than a regression.

Both symptoms share one shape: a presented alert reports no actions, and the assertion times out through ALERT_DEADLINE_EXCEEDED.

Separate symptom on the same step family, worth tracking apart from the two above: main@8e8eeb2ced (run 34712516058) failed Run fixture-backed iOS simulator E2E smoke with an assertion raised at test/integration/ios-simulator-e2e/live-automation-scenario.ts:33 (observeFixtureHome).

Required behavior

  • The alert-observation assertions in apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift pass or fail on their own terms, not on simulator alert presentation timing.
  • A presented system alert that the test can see must expose its button list to the probe, or the probe must report the empty-button case as a distinct typed outcome instead of consuming the whole deadline.
  • Whatever the resolution, an unchanged main must not require a manual re-run to go green.

Observable completion conditions

  • A diagnosis recorded here naming the mechanism (alert not yet realized when the action list is read, deadline sized below host latency, or a genuine runner-side ordering bug).
  • If it is timing: the probe or deadline change lands and the iOS workflow's Smoke Tests job is green across at least five consecutive main pushes, with no manual re-runs.
  • If it is a real defect: a failing-then-passing XCTest that pins it, plus the same five-run streak.
  • The observeFixtureHome case is either fixed or split into its own issue with the above evidence linked.

Dependencies

  • Reproduce on a hosted macOS runner or a local simulator; the failure does not appear in unit or check:affected runs, so this is device-facing and docs/agents/device-verification.md applies.
  • Related history for the alert horizon: ADR-tracked alert deadline work behind testAlertResolutionCannotBypassRequestedDeadline and testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched, which pass in the same runs where the two above fail.

Discovered while babysitting #2533, whose diff touched no Apple source.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageNew or unreviewed issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions