Make coding agent voice aware for better voice experience - #328217
Merged
Megan Rogge (meganrogge) merged 8 commits intoJul 31, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end voice awareness for coding-agent requests, enabling concise spoken progress and richer confirmation narration.
Changes:
- Propagates Voice Mode metadata through chat APIs and Copilot prompts.
- Adds transient progress checkpoints and narration protocol handling.
- Improves confirmation narration, voice approvals, accessibility help, and tests.
Reviewed changes
Copilot reviewed 42 out of 44 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/vscode-dts/vscode.proposed.chatParticipantPrivate.d.ts |
Defines private voice request and progress APIs. |
src/vs/workbench/contrib/chat/test/common/widget/annotations.test.ts |
Tests hidden progress rendering. |
src/vs/workbench/contrib/chat/test/common/model/chatModel.test.ts |
Tests transient metadata and serialization. |
src/vs/workbench/contrib/chat/test/common/chatService/chatService.test.ts |
Tests request metadata propagation. |
src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceToolDispatchService.test.ts |
Tests voice confirmation dispatch. |
src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts |
Tests narration protocol metadata. |
src/vs/workbench/contrib/chat/test/browser/accessibility/chatAccessibilityHelp.test.ts |
Tests agent-specific help text. |
src/vs/workbench/contrib/chat/common/widget/annotations.ts |
Excludes voice progress from rendering. |
src/vs/workbench/contrib/chat/common/voiceClient/voiceConfirmation.ts |
Classifies pending confirmations. |
src/vs/workbench/contrib/chat/common/voiceClient/voiceClientService.ts |
Extends voice protocol types. |
src/vs/workbench/contrib/chat/common/participants/chatAgents.ts |
Adds voice request metadata. |
src/vs/workbench/contrib/chat/common/model/chatSessionOperationLog.ts |
Excludes progress from persistence. |
src/vs/workbench/contrib/chat/common/model/chatModel.ts |
Models transient voice progress. |
src/vs/workbench/contrib/chat/common/chatService/chatServiceImpl.ts |
Forwards voice metadata to participants. |
src/vs/workbench/contrib/chat/common/chatService/chatService.ts |
Defines progress and request types. |
src/vs/workbench/contrib/chat/browser/widgetHosts/viewPane/chatViewPane.ts |
Marks pane voice submissions. |
src/vs/workbench/contrib/chat/browser/widget/chatWidget.ts |
Passes voice submission options. |
src/vs/workbench/contrib/chat/browser/voiceClient/voiceToolDispatchService.ts |
Handles typed confirmation approval. |
src/vs/workbench/contrib/chat/browser/voiceClient/voiceTelemetry.ts |
Updates narration classifications. |
src/vs/workbench/contrib/chat/browser/voiceClient/voiceSessionController.ts |
Orchestrates progress and confirmation narration. |
src/vs/workbench/contrib/chat/browser/voiceClient/voiceClientService.ts |
Implements extended narration protocol. |
src/vs/workbench/contrib/chat/browser/chat.ts |
Extends widget submission options. |
src/vs/workbench/contrib/chat/browser/actions/chatAccessibilityHelp.ts |
Documents spoken agent progress. |
src/vs/workbench/api/test/common/extHostTypeConverters.test.ts |
Tests progress conversion. |
src/vs/workbench/api/common/extHostTypes.ts |
Implements the progress response part. |
src/vs/workbench/api/common/extHostTypeConverters.ts |
Converts voice API values. |
src/vs/workbench/api/common/extHostChatAgents2.ts |
Exposes progress streaming. |
src/vs/workbench/api/common/extHost.api.impl.ts |
Registers the proposed class. |
extensions/copilot/src/vscodeTypes.ts |
Re-exports the progress class. |
extensions/copilot/src/util/common/test/shims/vscodeTypesShim.ts |
Adds progress to the test shim. |
extensions/copilot/src/util/common/test/shims/chatTypes.ts |
Defines the shim progress class. |
extensions/copilot/src/util/common/chatResponseStreamImpl.ts |
Implements progress streaming. |
extensions/copilot/src/extension/test/node/pseudoStartStopConversationCallback.spec.ts |
Tests hidden tool-card behavior. |
extensions/copilot/src/extension/prompts/node/agent/test/agentPrompt.spec.tsx |
Tests voice-only prompt guidance. |
extensions/copilot/src/extension/prompts/node/agent/agentPrompt.tsx |
Adds spoken-progress instructions. |
extensions/copilot/src/extension/prompt/node/test/defaultIntentRequestHandler.spec.ts |
Tests handler-level fallback progress. |
extensions/copilot/src/extension/prompt/node/pseudoStartStopConversationCallback.ts |
Hides internal progress tools. |
extensions/copilot/src/extension/prompt/node/defaultIntentRequestHandler.ts |
Enables progress for Agent intent. |
extensions/copilot/src/extension/linkify/common/responseStreamWithLinkification.ts |
Forwards progress through linkification. |
extensions/copilot/src/extension/intents/test/node/toolCallingLoopAutopilot.spec.ts |
Tests progress tool behavior. |
extensions/copilot/src/extension/intents/node/toolCallingLoop.ts |
Implements progress tooling and fallbacks. |
extensions/copilot/src/extension/codeBlocks/node/codeBlockProcessor.ts |
Forwards progress through code tracking. |
extensions/copilot/src/extension/chatSessions/copilotcli/node/copilotcliSession.ts |
Routes CLI progress responses. |
Comments suppressed due to low confidence (1)
src/vs/workbench/contrib/chat/browser/voiceClient/voiceSessionController.ts:3747
- This logs the raw extension-supplied progress text. The proposed API accepts arbitrary strings, so a third-party/private participant—or a model summary that bypasses the heuristic filter—can place sensitive workspace content into exported trace logs. Log only the narration ID/stage/sequence here.
this.logService.trace(`[voice][checkpoint] requested narration_id=${narrationId} request_id=${checkpoint?.requestId ?? '<unknown>'} phase=${checkpoint?.checkpointId ?? '<unknown>'} sequence=${checkpoint?.sequence ?? 0} seed=${JSON.stringify(text)}`);
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| this._pendingNarrationRetries.clear(); | ||
| this._voiceProgressListeners.clearAndDisposeAll(); | ||
| this._voiceProgressSessionByResponse.clear(); | ||
| this._lastSpokenAtBySession.clear(); |
Collaborator
There was a problem hiding this comment.
Could we also reset _autoListenSuppressed , _activeSessionShown , and _externalActiveSessionMode during fatal teardown, as disconnect() does? Otherwise reconnect can retain suppressed hands-free listening or stale session routing.
Mir (mirimadahmed)
force-pushed
the
mirimadahmed-microsoft-voice-aware-agent-harness
branch
from
July 30, 2026 15:25
f9d9f0a to
fd361a7
Compare
Co-authored-by: Copilot App <[email protected]>
Co-authored-by: Copilot App <[email protected]>
Mir (mirimadahmed)
force-pushed
the
mirimadahmed-microsoft-voice-aware-agent-harness
branch
from
July 30, 2026 15:26
fd361a7 to
2e2f093
Compare
Collaborator
|
how does it work if the session is not focused? do we just discard the voice responses? |
Contributor
Author
|
Intermediate progress is ephemeral. If its session isn’t shown when the checkpoint becomes eligible, it’s discarded. If checkpoint audio arrives after navigating away, it’s dropped rather than buffered. Returning only replays sticky final responses and confirmations, never old progress updates. |
Megan Rogge (meganrogge)
enabled auto-merge (squash)
July 30, 2026 18:14
Megan Rogge (meganrogge)
previously approved these changes
Jul 30, 2026
Megan Rogge (meganrogge)
previously approved these changes
Jul 30, 2026
Collaborator
|
Copilot resolve the merge conflicts in this pull request |
Resolve the chat model test conflict by preserving both voice progress and usage refinement coverage. Co-authored-by: Copilot <[email protected]> Copilot-Session: 1f93d2f2-660e-46e9-a869-98ce147905f9
Co-authored-by: Copilot <[email protected]> Copilot-Session: 1f93d2f2-660e-46e9-a869-98ce147905f9
Dmitriy Vasyura (dmitrivMS)
previously approved these changes
Jul 30, 2026
Co-authored-by: Copilot <[email protected]> Copilot-Session: 1f93d2f2-660e-46e9-a869-98ce147905f9
Dmitriy Vasyura (dmitrivMS)
previously approved these changes
Jul 30, 2026
roblourens
previously approved these changes
Jul 30, 2026
Co-authored-by: Copilot <[email protected]> Copilot-Session: d5f00f75-49d2-4b4f-a503-050b7b6a2f57
Megan Rogge (meganrogge)
approved these changes
Jul 31, 2026
Megan Rogge (meganrogge)
approved these changes
Jul 31, 2026
Vritant Bhardwaj (vritant24)
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.