Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Finalize telemetry at invocation completion - #14390

Merged
BagToad merged 15 commits into
bagtoad/attachment-operation-telemetryfrom
williammartin-wm-invocation-telemetry
Sep 9, 2026
Merged

Finalize telemetry at invocation completion#14390
BagToad merged 15 commits into
bagtoad/attachment-operation-telemetryfrom
williammartin-wm-invocation-telemetry

Conversation

@williammartin

@williammartin williammartin commented Sep 8, 2026

Copy link
Copy Markdown
Member

Builds on #14360; review against that branch to isolate this alternative.

Description

Keep one attachment event, without calling back into command state during telemetry shutdown.

#14360 combines the number of supplied attachments with the Markdown operations completed by their uploads. Those facts become available at different times. Its RecordDeferred callback builds the final event during Flush.

This alternative records the facts as they become known. The telemetry service copies them into a pending event; Main still owns completion, using one deferred Finish() to snapshot and send the payload. Attachment commands use a small, typed interface:

opts.AttachEvent = attachments.BeginTelemetry(
	telemetry, cmd.CommandPath(), opts.AttachFlag.Count(),
)
opts.Assets, err = opts.AttachFlag.UserAssets()
// Handle validation errors.

// After UploadAndAttach, before handling any upload error:
opts.AttachEvent.RecordOperations(uploadResult)

Intentional difference from #14360: attachment telemetry begins immediately before file validation, not in Args. Supplying 51 attachments still records attach_count=51 when the 50-file limit rejects them. Authentication, positional-argument, and other failures before UserAssets() do not produce an attachment event or promote attachment sampling. Generic command_invocation telemetry now covers those earlier failures.

How did you test this change?

I built and recorded gh at 38704130e876 with GH_TELEMETRY=log, isolated configuration, and dummy credentials. The recordings show real commands; the visible sed/jq pipeline selects fields from their actual telemetry logs.

The upload/write scenarios use a local, non-forwarding simulated GitHub API, reached through isolated http_unix_socket configuration. They exercise the real client and Markdown transformation, not GitHub's production upload or comment service.

Rejected input still counts

I supplied 51 attachment values to gh issue comment. It exited with the 50-attachment limit error, while recording attach_count=51, zero operations, and sample_rate=100. This case needs no API calls.

Real gh rejects 51 attachments while telemetry retains count 51, zero operations, and full sampling

Two references are one replacement

I supplied two PNGs, referenced the first twice in the body, and left the second unreferenced. I saw both references rewritten and the second image appended in the actual outgoing comment request. Telemetry recorded two attachments, one replacement, and one append at full sampling.

Real gh against a local simulated API rewrites two references to one image, appends another image, and records two attachments, one replacement, and one append

I also injected failures into the same local fixture:

Scenario I ran Result I observed Attach / replace / append
First upload succeeds; second fails First image's replacements retained in the comment request; comment write succeeds, but gh exits 1 for the upload failure 2 / 1 / 0
Both uploads succeed; comment write fails gh exits 1; completed Markdown operations remain in telemetry 2 / 1 / 1

Both failure cases retained the paired command event and full sampling. Separately, during development I exercised all six commands with positional errors and earlier flag conflicts, plus an unauthenticated issue-comment invocation: each logged only the generic command event, not an attachment event.

Authorship and follow-up

Who wrote this:

  • A human wrote it.
  • An agent wrote it under close human direction.
  • An agent wrote it independently, and no human has guided the implementation beyond the initial prompt.

Who answers review comments:

  • @williammartin will read and reply directly.
  • An agent will draft replies and @username will read them before they are posted.
  • Nobody has explicitly committed to replying.

Copilot AI balanced review requested due to automatic review settings September 8, 2026 14:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The broad telemetry lifecycle and command behavior changes require final human review.

Review tier: Balanced
Findings: None

What changed in this PR

Reworks telemetry to separate invocation-owned event collection and completion from payload delivery while preserving attachment metrics and sampling behavior.

Changes:

  • Adds pending events and explicit invocation completion.
  • Finalizes telemetry before delivery, including on early failures.
  • Updates command, attachment, and skills telemetry integrations and tests.
File Description
pkg/​cmdutil/​telemetry.go Starts command events before argument validation.
pkg/​cmdutil/​telemetry_test.go Tests expanded command lifecycle coverage.
pkg/​cmd/​skills/​skills.go Uses invocation telemetry and sampling.
pkg/​cmd/​skills/​skills_test.go Updates the sampling test double.
pkg/​cmd/​skills/​search/​search.go Narrows the telemetry dependency.
pkg/​cmd/​skills/​search/​search_test.go Updates telemetry fakes.
pkg/​cmd/​skills/​preview/​preview.go Narrows the telemetry dependency.
pkg/​cmd/​skills/​preview/​preview_test.go Updates telemetry fakes.
pkg/​cmd/​skills/​list/​list.go Narrows the telemetry dependency.
pkg/​cmd/​skills/​list/​list_test.go Updates event recorder tests.
pkg/​cmd/​skills/​install/​install.go Narrows the telemetry dependency.
pkg/​cmd/​skills/​install/​install_test.go Updates telemetry fakes.
pkg/​cmd/​root/​root.go Passes invocation telemetry through the command tree.
pkg/​cmd/​root/​help_test.go Uses no-op invocation telemetry.
pkg/​cmd/​root/​extension_registration_test.go Uses no-op invocation telemetry.
pkg/​cmd/​pr/​shared/​commentable_test.go Finalizes attachment telemetry in tests.
pkg/​cmd/​pr/​pr.go Accepts invocation telemetry.
pkg/​cmd/​pr/​edit/​edit.go Accepts invocation telemetry.
pkg/​cmd/​pr/​edit/​edit_test.go Updates attachment telemetry tests.
pkg/​cmd/​pr/​create/​create.go Accepts invocation telemetry.
pkg/​cmd/​pr/​create/​create_test.go Updates attachment telemetry tests.
pkg/​cmd/​pr/​comment/​comment.go Accepts invocation telemetry.
pkg/​cmd/​pr/​comment/​comment_test.go Updates attachment telemetry tests.
pkg/​cmd/​issue/​issue.go Accepts invocation telemetry.
pkg/​cmd/​issue/​edit/​edit.go Accepts invocation telemetry.
pkg/​cmd/​issue/​edit/​edit_test.go Updates attachment telemetry tests.
pkg/​cmd/​issue/​create/​create.go Accepts invocation telemetry.
pkg/​cmd/​issue/​create/​create_test.go Updates attachment telemetry tests.
pkg/​cmd/​issue/​comment/​comment.go Accepts invocation telemetry.
pkg/​cmd/​issue/​comment/​comment_test.go Updates attachment telemetry tests.
pkg/​cmd/​factory/​default_test.go Uses no-op invocation telemetry.
pkg/​cmd/​copilot/​copilot.go Uses invocation-wide telemetry policy.
pkg/​cmd/​copilot/​copilot_test.go Updates the telemetry test double.
pkg/​cmd/​attestation/​verify/​verify_integration_test.go Uses no-op invocation telemetry.
pkg/​cmd/​api/​api_test.go Tests explicit completion and delivery.
internal/​telemetry/​telemetry.go Removes the combined telemetry service.
internal/​telemetry/​telemetry_test.go Tests invocation and delivery behavior.
internal/​telemetry/​invocation.go Implements invocation-owned event collection.
internal/​telemetry/​invocation_test.go Tests snapshots, sampling, and concurrency.
internal/​telemetry/​fake.go Adds lifecycle-aware telemetry spies.
internal/​telemetry/​delivery.go Adds completed-payload delivery buffering.
internal/​ghcmd/​cmd.go Defers completion before delivery.
internal/​gh/​ghtelemetry/​telemetry.go Defines pending-event and invocation interfaces.
internal/​attachments/​test.go Updates attachment telemetry helpers.
internal/​attachments/​telemetry.go Updates pending attachment events through handles.
internal/​attachments/​flags_test.go Tests attachment event completion.
cmd/​gen-docs/​main.go Uses no-op invocation telemetry.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

williammartin and others added 13 commits September 8, 2026 16:36
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 9452b3e7-02d8-4d4d-b2db-ab18b35657c1
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: c95db6c5-fc17-4880-915c-5d2c4b938d49
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: c95db6c5-fc17-4880-915c-5d2c4b938d49
},
{
name: "--attach telemetry survives argument validation",
name: "argument validation skips attachment telemetry",

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BagToad think this is the biggest question mark on this PR. Moving BeginTelemetry into RunE drops attachment_invocation when execution stops during:

  • Positional argument validation
    • Parent PersistentPreRunE, including authentication failures
  • Command PreRunE
  • Early RunE validation before UserAssets(), such as incompatible flags or missing required non-interactive input

Moving it into the command’s PreRunE would recover the latter two cases, while still excluding positional-validation and authentication failures.

How valuable are those earlier failures in practice, especially without an outcome or error dimension? If this event is interpreted as attachment usage, failed attempts may overstate successful adoption. I'm just not sure what you're looking for here.

I ask because wrapping Args is not an established pattern here, and using an argument-validation hook to begin telemetry feels like the wrong seam to commit to solely to capture those failures.

This comment was marked as low quality.

@williammartin
williammartin marked this pull request as ready for review September 9, 2026 10:25
@williammartin
williammartin requested review from a team as code owners September 9, 2026 10:25
@williammartin
williammartin requested review from BagToad and tidy-dev and removed request for tidy-dev September 9, 2026 10:25
@BagToad
BagToad merged commit ae1ba32 into trunk Sep 9, 2026
25 checks passed
@BagToad
BagToad deleted the williammartin-wm-invocation-telemetry branch September 9, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants