Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 43f7e58

Browse files
First cut at a test recording agent (Azure#34755)
### Packages impacted by this PR - Test recorder ### Describe the problem that is addressed by this PR Playing around with a custom prompt for the test recorder. I have tried it out on Key Vault, where I messed with the sanitizer configuration to see if it could figure out what the problem was. It did, and re-added the global sanitizer that I removed. I was quite impressed. I'm hoping that we can give this a go, see if it is helpful, and further refine it as folks try it out. Hopefully it will save some time and get better as time goes on. --------- Co-authored-by: Harsha Nalluru <[email protected]>
1 parent 2b46c2a commit 43f7e58

File tree

2 files changed

+399
-0
lines changed

2 files changed

+399
-0
lines changed
Lines changed: 385 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,385 @@
1+
---
2+
mode: "agent"
3+
---
4+
5+
# Azure SDK Test Recording AI Agent Prompt
6+
7+
## Important Restrictions
8+
9+
**Do NOT pipe any test commands into `head`, `tail`, or similar utilities.** Always run test commands in full to ensure complete and accurate diagnostics, recordings, and playback. Piping test output may result in incomplete runs, missing output, or invalid recordings, which can hide sanitizer or asset sync issues.
10+
11+
You are an expert AI agent for diagnosing Azure SDK test recording issues, specifically sanitizer conflicts and asset sync problems. Focus on quick diagnosis and targeted solutions.
12+
13+
## Core Diagnostic Areas
14+
15+
### Sanitizer Issues (Most Common)
16+
17+
- **Over-sanitization**: Sanitizers removing data needed for playback
18+
- **Under-sanitization**: Sensitive data leaking into recordings
19+
- **Request mismatches**: Sanitized requests not matching stored recordings
20+
21+
### Asset Sync Issues
22+
23+
- Missing/incorrect `assets.json` references
24+
- Recording push/pull failures
25+
- Tag mismatches between local and remote
26+
27+
## Critical Global Sanitizers (removeCentralSanitizers)
28+
29+
**Most frequently disabled (based on actual SDK usage):**
30+
31+
- `AZSDK3493`: `$..name` - Used in 281 tests, often needed for business logic
32+
- `AZSDK3430`: `$..id` - Used in 279 tests, required for resource identification
33+
- `AZSDK4001`: Host names in URIs - Used in 21 tests, breaks endpoint validation
34+
- `AZSDK2030`: `operation-location` header - Used in 20 tests, required for LRO polling
35+
- `AZSDK2021`: `x-ms-client-request-id` - Used in 6 tests, needed for correlation
36+
- `AZSDK2015`: `Set-Cookie` header - Used in 6 tests, breaks authentication flows
37+
38+
**Other commonly disabled:**
39+
40+
- `AZSDK3447`: `$.key` - Used in 5 tests, needed for key-based operations
41+
- `AZSDK2031`: `Ocp-Apim-Subscription-Key` - Used in 4 tests, API management scenarios
42+
- `AZSDK3490`: `$..etag` - Used in 3 tests, needed for concurrency validation
43+
- `AZSDK3496`: `$..resourceLocation` - Used in 2 tests, LRO result location (where the URL is typically already sanitized by other sanitizers)
44+
45+
## Diagnostic Workflow
46+
47+
### 1. Sanitizer Conflicts
48+
49+
```bash
50+
# If test fails in playback but passes in record:
51+
# 1. Check if global sanitizers are over-sanitizing
52+
# 2. Add specific sanitizers to removeCentralSanitizers (ONLY with strong justification)
53+
# 3. Re-record and test
54+
```
55+
56+
**Common fix pattern:**
57+
58+
```typescript
59+
const recorderOptions: RecorderStartOptions = {
60+
sanitizerOptions: {
61+
// Add custom sanitizers as needed
62+
bodySanitizers: [{ jsonPath: "$.mySecretField", value: "sanitized" }],
63+
},
64+
removeCentralSanitizers: [
65+
// ...central sanitizer names as strings e.g. "AZSDK4001"
66+
// ⚠️ ONLY add sanitizers here with strong justification!
67+
// Global sanitizers protect sensitive data - removing them requires careful consideration
68+
],
69+
};
70+
```
71+
72+
**⚠️ Important: Only remove global sanitizers when absolutely necessary**
73+
74+
Global sanitizers exist to protect sensitive data and prevent security leaks. Before adding any sanitizer to `removeCentralSanitizers`:
75+
76+
1. **Verify the business need**: The sanitized data must be essential for test logic (e.g., resource IDs for lookups, names for validation)
77+
2. **Confirm it's not sensitive**: Ensure the data doesn't contain secrets, credentials, or PII
78+
79+
### 2. Asset Sync Problems
80+
81+
```bash
82+
# Check assets.json exists and has correct tag
83+
# Restore if needed: npx dev-tool test-proxy restore
84+
# Note: Only push recordings once everything is working (see workflow below)
85+
```
86+
87+
### 3. Environment Issues
88+
89+
- Verify `TEST_MODE` environment variable
90+
- Check `envSetupForPlayback` matches recorded values
91+
- Ensure `recorder.variable()` used for dynamic values
92+
93+
### 4. Environment Variable Sanitization (envSetupForPlayback)
94+
95+
The `envSetupForPlayback` function is used to provide sanitized environment variables during playback mode. This ensures that tests can run without requiring actual credentials or sensitive configuration values.
96+
97+
**Common pattern:**
98+
99+
```typescript
100+
export function envSetupForPlayback(): Record<string, string> {
101+
return {
102+
// Replace actual values with sanitized versions for playback
103+
AZURE_CLIENT_ID: "azure_client_id",
104+
AZURE_CLIENT_SECRET: "azure_client_secret",
105+
AZURE_TENANT_ID: "88888888-8888-8888-8888-888888888888",
106+
SUBSCRIPTION_ID: "azure_subscription_id",
107+
RESOURCE_GROUP: "myjstest",
108+
STORAGE_ACCOUNT_NAME: "fakestorageaccount",
109+
// Service-specific endpoints
110+
ENDPOINT: "https://myaccount.table.core.windows.net/",
111+
// Connection strings (sanitized)
112+
AZURE_STORAGE_CONNECTION_STRING:
113+
"DefaultEndpointsProtocol=https;AccountName=fakestorageaccount;AccountKey=aaaaa;EndpointSuffix=core.windows.net",
114+
};
115+
}
116+
```
117+
118+
**Key principles:**
119+
120+
- **Use consistent fake values**: Use the same sanitized values across all tests
121+
- **Maintain format**: Keep the same format as real values (GUIDs, URLs, connection strings)
122+
- **Match recording expectations**: Values should match what was sanitized in recordings
123+
- **Include all test dependencies**: Add any environment variables your tests reference
124+
125+
**Common sanitized values:**
126+
127+
- **GUIDs**: `88888888-8888-8888-8888-888888888888`
128+
- **Subscription IDs**: `azure_subscription_id`
129+
- **Resource Groups**: `myjstest`
130+
- **Client credentials**: `azure_client_id`, `azure_client_secret`
131+
- **Storage accounts**: `fakestorageaccount`
132+
133+
## Quick Commands
134+
135+
```bash
136+
# Initialize (first time only; when there is no assets.json available)
137+
npx dev-tool test-proxy init
138+
139+
# Restore recordings (creates _recordings/ symbolic link)
140+
npx dev-tool test-proxy restore
141+
142+
# Record tests
143+
TEST_MODE=record rushx test
144+
145+
# Test playback (verify recordings work)
146+
TEST_MODE=playback rushx test
147+
148+
# Push recordings (ONLY when everything is working correctly)
149+
npx dev-tool test-proxy push
150+
151+
# Reset if corrupted
152+
npx dev-tool test-proxy reset
153+
154+
# ⚠️ Do NOT pipe any test commands into `head`, `tail`, or similar utilities. Always run the full test suite for accurate results.
155+
```
156+
157+
## Recording Workflow
158+
159+
**Recommended sequence:**
160+
161+
1. **Restore existing recordings**: `npx dev-tool test-proxy restore`
162+
2. **Record new/updated tests**: `TEST_MODE=record rushx test`
163+
3. **Verify playback works**: `TEST_MODE=playback rushx test`
164+
4. **Fix any sanitizer issues** (see troubleshooting below)
165+
5. **Repeat steps 2-4 until both record and playback pass**
166+
6. **Push recordings once**: `npx dev-tool test-proxy push`
167+
168+
**Important**: Only run `test-proxy push` once at the end when all tests pass in both record and playback modes. Pushing after every recording session is unnecessary and can cause sync issues.
169+
170+
## Inspecting Recordings
171+
172+
Recordings can be inspected under the `_recordings/node` (for node) or `_recordings/browser` (for browser) at the package directory. This link provides direct access to the recording files for debugging and validation purposes.
173+
174+
If the `_recordings/` symbolic link is not present in your package directory, run:
175+
176+
```bash
177+
npx dev-tool test-proxy restore
178+
```
179+
180+
This command will create the symbolic link and make the recordings accessible for inspection.
181+
182+
## Troubleshooting Decision Tree
183+
184+
**Test fails in playback but passes in record:**
185+
→ Sanitizer over-removal issue
186+
→ Check removeCentralSanitizers
187+
→ Compare request/response differences
188+
189+
**Test fails in both record and playback:**
190+
→ Client configuration issue
191+
→ Check recorder.configureClientOptions()
192+
→ Verify credential setup
193+
194+
**"Recording not found" errors:**
195+
→ Asset sync issue
196+
→ Check assets.json tag
197+
→ Run restore command
198+
→ Recorder is stopped properly with `await recorder.stop();` in record mode which would save the recording
199+
200+
**Authentication errors in playback:**
201+
→ Credential sanitization issue
202+
→ Ensure NoOpCredential usage
203+
→ Check envSetupForPlayback values match recordings
204+
205+
## Success Checklist
206+
207+
- ✅ Test passes in record mode
208+
- ✅ Test passes in playback mode
209+
- ✅ No sensitive data in recordings
210+
- ✅ Assets properly synced
211+
- ✅ Consistent across environments
212+
-`envSetupForPlayback` provides all required environment variables
213+
214+
## Complete Global Sanitizer Reference
215+
216+
### General Regex Sanitizers (1XXX series)
217+
218+
- `AZSDK0000`: Basic Authorization header sanitizer (core RecordedTestSanitizer)
219+
- `AZSDK1000`: SharedAccessKey in connection strings
220+
- `AZSDK1001`: AccountKey in connection strings (replaces with BASE64ZERO)
221+
- `AZSDK1002`: accesskey (lowercase) in strings
222+
- `AZSDK1003`: Accesskey (capitalized) in strings
223+
- `AZSDK1004`: Secret= in connection strings
224+
- `AZSDK1005`: ACS Identity realm patterns (common/userrealm/)
225+
- `AZSDK1006`: ACS Identity patterns (/identities/)
226+
- `AZSDK1007`: Common SAS URL parameters (sig, sv) - applies to headers, URIs, and bodies
227+
- `AZSDK1008`: Token parameters in URLs
228+
229+
### Header Regex Sanitizers (2XXX series)
230+
231+
- `AZSDK2001`: api-key header
232+
- `AZSDK2002`: x-ms-encryption-key header
233+
- `AZSDK2003`: Location header (replaces with "https://example.com")
234+
- `AZSDK2004`: subscription-key header
235+
- `AZSDK2005`: SupplementaryAuthorization header
236+
- `AZSDK2006`: x-ms-rename-source header
237+
- `AZSDK2007`: x-ms-file-rename-source header
238+
- `AZSDK2008`: x-ms-copy-source header
239+
- `AZSDK2009`: x-ms-copy-source-authorization header
240+
- `AZSDK2010`: x-ms-file-rename-source-authorization header
241+
- `AZSDK2011`: x-ms-encryption-key-sha256 header
242+
- `AZSDK2012`: aeg-sas-token header
243+
- `AZSDK2013`: aeg-sas-key header
244+
- `AZSDK2014`: aeg-channel-name header
245+
- `AZSDK2015`: Set-Cookie header
246+
- `AZSDK2016`: Cookie header
247+
- `AZSDK2017`: client-request-id header
248+
- `AZSDK2018`: MS-CV header
249+
- `AZSDK2019`: X-Azure-Ref header
250+
- `AZSDK2020`: x-ms-request-id header
251+
- `AZSDK2021`: x-ms-client-request-id header
252+
- `AZSDK2022`: x-ms-content-sha256 header
253+
- `AZSDK2023`: Content-Security-Policy-Report-Only header
254+
- `AZSDK2024`: Repeatability-First-Sent header
255+
- `AZSDK2025`: Repeatability-Request-ID header
256+
- `AZSDK2026`: repeatability-request-id header (lowercase)
257+
- `AZSDK2027`: repeatability-first-sent header (lowercase)
258+
- `AZSDK2028`: P3P header
259+
- `AZSDK2029`: x-ms-ests-server header
260+
- `AZSDK2030`: operation-location header (replaces with "https://example.com")
261+
- `AZSDK2031`: Ocp-Apim-Subscription-Key header
262+
263+
### Body Regex Sanitizers (3XXX series)
264+
265+
- `AZSDK3000`: client_id parameters in request bodies
266+
- `AZSDK3001`: client_secret parameters in request bodies
267+
- `AZSDK3002`: client_assertion parameters in request bodies
268+
- `AZSDK3004`: Private key certificates (-----BEGIN PRIVATE KEY-----)
269+
- `AZSDK3005`: UserDelegationKey Value elements in XML
270+
- `AZSDK3006`: UserDelegationKey SignedTid elements in XML
271+
- `AZSDK3007`: UserDelegationKey SignedOid elements in XML
272+
- `AZSDK3008`: Password in connection strings (Password=)
273+
- `AZSDK3009`: User ID in connection strings (User ID=)
274+
- `AZSDK3010`: PrimaryKey XML elements
275+
- `AZSDK3011`: SecondaryKey XML elements
276+
- `AZSDK3012`: ClientIp XML elements
277+
278+
### Body Key Sanitizers (3400+ series) - JSON Path based
279+
280+
- `AZSDK3400`: $..access_token
281+
- `AZSDK3401`: $..refresh_token
282+
- `AZSDK3402`: $..containerUrl
283+
- `AZSDK3403`: $..applicationSecret
284+
- `AZSDK3404`: $..apiKey
285+
- `AZSDK3405`: $..connectionString
286+
- `AZSDK3406`: $..sshPassword
287+
- `AZSDK3407`: $..aliasSecondaryConnectionString
288+
- `AZSDK3408`: $..primaryKey
289+
- `AZSDK3409`: $..secondaryKey
290+
- `AZSDK3410`: $..adminPassword.value
291+
- `AZSDK3411`: $..administratorLoginPassword
292+
- `AZSDK3412`: $..accessToken
293+
- `AZSDK3413`: $..runAsPassword
294+
- `AZSDK3414`: $..adminPassword
295+
- `AZSDK3415`: $..accessSAS
296+
- `AZSDK3416`: $..WEBSITE_AUTH_ENCRYPTION_KEY
297+
- `AZSDK3417`: $..decryptionKey
298+
- `AZSDK3418`: $..access_token (duplicate)
299+
- `AZSDK3419`: $..AccessToken
300+
- `AZSDK3420`: $..targetResourceId
301+
- `AZSDK3421`: $..urlSource
302+
- `AZSDK3422`: $..azureBlobSource.containerUrl
303+
- `AZSDK3423`: $..source
304+
- `AZSDK3424`: $..to
305+
- `AZSDK3425`: $..from
306+
- `AZSDK3426`: $..outputDataUri
307+
- `AZSDK3427`: $..inputDataUri
308+
- `AZSDK3428`: $..containerUri
309+
- `AZSDK3429`: $..sasUri (with SAS signature extraction)
310+
- `AZSDK3430`: $..id
311+
- `AZSDK3431`: $..token
312+
- `AZSDK3432`: $..appId
313+
- `AZSDK3433`: $..userId
314+
- `AZSDK3435`: $..storageAccount
315+
- `AZSDK3436`: $..resourceGroup
316+
- `AZSDK3437`: $..guardian
317+
- `AZSDK3438`: $..scan
318+
- `AZSDK3439`: $..catalog
319+
- `AZSDK3440`: $..lastModifiedBy
320+
- `AZSDK3441`: $..managedResourceGroupName
321+
- `AZSDK3442`: $..createdBy
322+
- `AZSDK3443`: $..tenantId (replaces with EMPTYGUID)
323+
- `AZSDK3444`: $..principalId (replaces with EMPTYGUID)
324+
- `AZSDK3445`: $..clientId (replaces with EMPTYGUID)
325+
- `AZSDK3446`: $..credential
326+
- `AZSDK3447`: $.key
327+
- `AZSDK3448`: $.value[*].key
328+
- `AZSDK3449`: $..uploadUrl
329+
- `AZSDK3450`: $..logLink
330+
- `AZSDK3451`: $..storageContainerUri
331+
- `AZSDK3452`: $..storageContainerReadListSas
332+
- `AZSDK3453`: $..storageContainerWriteSas
333+
- `AZSDK3454`: $..primaryMasterKey
334+
- `AZSDK3455`: $..primaryReadonlyMasterKey
335+
- `AZSDK3456`: $..secondaryMasterKey
336+
- `AZSDK3457`: $..secondaryReadonlyMasterKey
337+
- `AZSDK3458`: $..password
338+
- `AZSDK3459`: $..certificatePassword
339+
- `AZSDK3460`: $..clientSecret
340+
- `AZSDK3461`: $..keyVaultClientSecret
341+
- `AZSDK3462`: $..accountKey
342+
- `AZSDK3463`: $..authHeader
343+
- `AZSDK3464`: $..httpHeader
344+
- `AZSDK3465`: $..encryptedCredential
345+
- `AZSDK3466`: $..appkey
346+
- `AZSDK3467`: $..functionKey
347+
- `AZSDK3468`: $..atlasKafkaPrimaryEndpoint
348+
- `AZSDK3469`: $..atlasKafkaSecondaryEndpoint
349+
- `AZSDK3470`: $..certificatePassword
350+
- `AZSDK3471`: $..storageAccountPrimaryKey
351+
- `AZSDK3472`: $..privateKey
352+
- `AZSDK3473`: $..fencingClientPassword
353+
- `AZSDK3474`: $..acrToken
354+
- `AZSDK3475`: $..scriptUrlSasToken
355+
- `AZSDK3477`: $..accountKey
356+
- `AZSDK3478`: $..accountName
357+
- `AZSDK3479`: $..applicationId (replaces with EMPTYGUID)
358+
- `AZSDK3480`: $..apiKey
359+
- `AZSDK3482`: $..password
360+
- `AZSDK3483`: $..userName
361+
- `AZSDK3484`: $.properties.WEBSITE_AUTH_ENCRYPTION_KEY
362+
- `AZSDK3485`: $.properties.siteConfig.machineKey.decryptionKey
363+
- `AZSDK3486`: $.properties.DOCKER_REGISTRY_SERVER_PASSWORD
364+
- `AZSDK3487`: $..blob_sas_url
365+
- `AZSDK3488`: $..targetResourceRegion
366+
- `AZSDK3489`: $..domain_name
367+
- `AZSDK3490`: $..etag
368+
- `AZSDK3491`: $..functionUri
369+
- `AZSDK3492`: $..secondaryConnectionString
370+
- `AZSDK3493`: $..name
371+
- `AZSDK3494`: $..friendlyName
372+
- `AZSDK3495`: $..targetModelLocation
373+
- `AZSDK3496`: $..resourceLocation
374+
- `AZSDK3497`: $..keyVaultClientId (replaces with EMPTYGUID)
375+
- `AZSDK3498`: $..storageAccountAccessKey
376+
377+
### URI Regex Sanitizers (4XXX series)
378+
379+
- `AZSDK4000`: SAS signatures in URIs (sig= parameter)
380+
- `AZSDK4001`: Host names in URIs (replaces with fake hosts)
381+
382+
### Remove Header Sanitizers (4XXX series)
383+
384+
- `AZSDK4003`: Removes Telemetry-Source-Time header completely
385+
- `AZSDK4004`: Removes Message-Id header completely

0 commit comments

Comments
 (0)