Finalize telemetry at invocation completion - #14390
Conversation
Co-authored-by: Copilot App <[email protected]>
Co-authored-by: Copilot App <[email protected]>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The broad telemetry lifecycle and command behavior changes require final human review.
Review tier: Balanced
Findings: None
What changed in this PR
Reworks telemetry to separate invocation-owned event collection and completion from payload delivery while preserving attachment metrics and sampling behavior.
Changes:
- Adds pending events and explicit invocation completion.
- Finalizes telemetry before delivery, including on early failures.
- Updates command, attachment, and skills telemetry integrations and tests.
| File | Description |
|---|---|
pkg/cmdutil/telemetry.go |
Starts command events before argument validation. |
pkg/cmdutil/telemetry_test.go |
Tests expanded command lifecycle coverage. |
pkg/cmd/skills/skills.go |
Uses invocation telemetry and sampling. |
pkg/cmd/skills/skills_test.go |
Updates the sampling test double. |
pkg/cmd/skills/search/search.go |
Narrows the telemetry dependency. |
pkg/cmd/skills/search/search_test.go |
Updates telemetry fakes. |
pkg/cmd/skills/preview/preview.go |
Narrows the telemetry dependency. |
pkg/cmd/skills/preview/preview_test.go |
Updates telemetry fakes. |
pkg/cmd/skills/list/list.go |
Narrows the telemetry dependency. |
pkg/cmd/skills/list/list_test.go |
Updates event recorder tests. |
pkg/cmd/skills/install/install.go |
Narrows the telemetry dependency. |
pkg/cmd/skills/install/install_test.go |
Updates telemetry fakes. |
pkg/cmd/root/root.go |
Passes invocation telemetry through the command tree. |
pkg/cmd/root/help_test.go |
Uses no-op invocation telemetry. |
pkg/cmd/root/extension_registration_test.go |
Uses no-op invocation telemetry. |
pkg/cmd/pr/shared/commentable_test.go |
Finalizes attachment telemetry in tests. |
pkg/cmd/pr/pr.go |
Accepts invocation telemetry. |
pkg/cmd/pr/edit/edit.go |
Accepts invocation telemetry. |
pkg/cmd/pr/edit/edit_test.go |
Updates attachment telemetry tests. |
pkg/cmd/pr/create/create.go |
Accepts invocation telemetry. |
pkg/cmd/pr/create/create_test.go |
Updates attachment telemetry tests. |
pkg/cmd/pr/comment/comment.go |
Accepts invocation telemetry. |
pkg/cmd/pr/comment/comment_test.go |
Updates attachment telemetry tests. |
pkg/cmd/issue/issue.go |
Accepts invocation telemetry. |
pkg/cmd/issue/edit/edit.go |
Accepts invocation telemetry. |
pkg/cmd/issue/edit/edit_test.go |
Updates attachment telemetry tests. |
pkg/cmd/issue/create/create.go |
Accepts invocation telemetry. |
pkg/cmd/issue/create/create_test.go |
Updates attachment telemetry tests. |
pkg/cmd/issue/comment/comment.go |
Accepts invocation telemetry. |
pkg/cmd/issue/comment/comment_test.go |
Updates attachment telemetry tests. |
pkg/cmd/factory/default_test.go |
Uses no-op invocation telemetry. |
pkg/cmd/copilot/copilot.go |
Uses invocation-wide telemetry policy. |
pkg/cmd/copilot/copilot_test.go |
Updates the telemetry test double. |
pkg/cmd/attestation/verify/verify_integration_test.go |
Uses no-op invocation telemetry. |
pkg/cmd/api/api_test.go |
Tests explicit completion and delivery. |
internal/telemetry/telemetry.go |
Removes the combined telemetry service. |
internal/telemetry/telemetry_test.go |
Tests invocation and delivery behavior. |
internal/telemetry/invocation.go |
Implements invocation-owned event collection. |
internal/telemetry/invocation_test.go |
Tests snapshots, sampling, and concurrency. |
internal/telemetry/fake.go |
Adds lifecycle-aware telemetry spies. |
internal/telemetry/delivery.go |
Adds completed-payload delivery buffering. |
internal/ghcmd/cmd.go |
Defers completion before delivery. |
internal/gh/ghtelemetry/telemetry.go |
Defines pending-event and invocation interfaces. |
internal/attachments/test.go |
Updates attachment telemetry helpers. |
internal/attachments/telemetry.go |
Updates pending attachment events through handles. |
internal/attachments/flags_test.go |
Tests attachment event completion. |
cmd/gen-docs/main.go |
Uses no-op invocation telemetry. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Co-authored-by: Copilot App <[email protected]> Copilot-Session: c95db6c5-fc17-4880-915c-5d2c4b938d49
Co-authored-by: Copilot App <[email protected]> Copilot-Session: c95db6c5-fc17-4880-915c-5d2c4b938d49
| }, | ||
| { | ||
| name: "--attach telemetry survives argument validation", | ||
| name: "argument validation skips attachment telemetry", |
There was a problem hiding this comment.
@BagToad think this is the biggest question mark on this PR. Moving BeginTelemetry into RunE drops attachment_invocation when execution stops during:
- Positional argument validation
-
- Parent
PersistentPreRunE, including authentication failures
- Parent
- Command
PreRunE - Early
RunEvalidation beforeUserAssets(), such as incompatible flags or missing required non-interactive input
Moving it into the command’s PreRunE would recover the latter two cases, while still excluding positional-validation and authentication failures.
How valuable are those earlier failures in practice, especially without an outcome or error dimension? If this event is interpreted as attachment usage, failed attempts may overstate successful adoption. I'm just not sure what you're looking for here.
I ask because wrapping Args is not an established pattern here, and using an argument-validation hook to begin telemetry feels like the wrong seam to commit to solely to capture those failures.
This comment was marked as low quality.
This comment was marked as low quality.
Sorry, something went wrong.
Builds on #14360; review against that branch to isolate this alternative.
Description
Keep one attachment event, without calling back into command state during telemetry shutdown.
#14360 combines the number of supplied attachments with the Markdown operations completed by their uploads. Those facts become available at different times. Its
RecordDeferredcallback builds the final event duringFlush.This alternative records the facts as they become known. The telemetry service copies them into a pending event;
Mainstill owns completion, using one deferredFinish()to snapshot and send the payload. Attachment commands use a small, typed interface:Intentional difference from #14360: attachment telemetry begins immediately before file validation, not in
Args. Supplying 51 attachments still recordsattach_count=51when the 50-file limit rejects them. Authentication, positional-argument, and other failures beforeUserAssets()do not produce an attachment event or promote attachment sampling. Genericcommand_invocationtelemetry now covers those earlier failures.How did you test this change?
I built and recorded
ghat38704130e876withGH_TELEMETRY=log, isolated configuration, and dummy credentials. The recordings show real commands; the visiblesed/jqpipeline selects fields from their actual telemetry logs.The upload/write scenarios use a local, non-forwarding simulated GitHub API, reached through isolated
http_unix_socketconfiguration. They exercise the real client and Markdown transformation, not GitHub's production upload or comment service.Rejected input still counts
I supplied 51 attachment values to
gh issue comment. It exited with the 50-attachment limit error, while recordingattach_count=51, zero operations, andsample_rate=100. This case needs no API calls.Two references are one replacement
I supplied two PNGs, referenced the first twice in the body, and left the second unreferenced. I saw both references rewritten and the second image appended in the actual outgoing comment request. Telemetry recorded two attachments, one replacement, and one append at full sampling.
I also injected failures into the same local fixture:
ghexits 1 for the upload failureghexits 1; completed Markdown operations remain in telemetryBoth failure cases retained the paired command event and full sampling. Separately, during development I exercised all six commands with positional errors and earlier flag conflicts, plus an unauthenticated issue-comment invocation: each logged only the generic command event, not an attachment event.
Authorship and follow-up
Who wrote this:
Who answers review comments: