feat(settings-validation): add validation for settings schema #12929
Conversation
|
@galz10 hi, When you have a moment, could you please take a look at this for me? |
packages/cli/src/config/settings.ts
Outdated
| return false; | ||
| } | ||
| // If a key exists that is both a V1 key and a V2 container (like 'model'), | ||
| // If a key exists that is 、 |
There was a problem hiding this comment.
Nit: please fix the comment
| error: z.ZodError, | ||
| filePath: string, | ||
| ): string { | ||
| const lines: string[] = []; |
There was a problem hiding this comment.
If validation fails inside an array (e.g., tools.allowed), the path will look like tools.allowed.0. A small helper to format the path would improve readability.
const path = issue.path.reduce((acc, curr) => {
return typeof curr === 'number' ? `${acc}[${curr}]` : `${acc ? acc + '.' : ''}${curr}`;
}, '');| import { z } from 'zod'; | ||
|
|
||
| describe('settings-validation', () => { | ||
| describe('validateSettings', () => { |
There was a problem hiding this comment.
There are no tests verifying that complex, nested configurations (like mcpServers or customThemes) actually fail when their internal structure is invalid. Ideally we should also have tests to validate complex setting structures.
| lines.push(''); | ||
|
|
||
| for (const issue of error.issues) { | ||
| const path = issue.path.join('.'); |
There was a problem hiding this comment.
If a user copies a large, completely malformed JSON block (or a mismatching schema version), they could trigger dozens of validation errors. A "wall of text" error message in the terminal can be overwhelming and push the useful context (the file path) off-screen. Limit the number of displayed errors in formatValidationError to the first 3-5 items, and append a "...and X more errors"
|
Got this feedback from gemini :
Ideally we shouldn't be crashing if the schema is incorrect, can you verify that we are not crashing if you have a old setting or a new setting that isn't in settingSchema |
|
@galz10 Unknown/Old Settings: I used .passthrough() in the Zod schema generation. This ensures that any extra keys (like deprecated settings or typos) are ignored and will not cause a crash or validation failure. Crash on Invalid Types: The previous behavior often allowed invalid types (e.g. a string for a boolean) to pass through, causing the CLI to crash unpredictably later during execution. My change catches these errors at startup. Currently, I've implemented a "fail fast" approach: if the configuration is strictly invalid (wrong types), it stops execution to prevent undefined behavior. Question: Do you prefer that I: A) Keep strict validation but ensure it exits gracefully with a friendly error message (no stack trace)? B) Log a warning, ignore the invalid config file, and proceed with system defaults? There is a specific case mentioned here, where configuring a field with an incorrect type could directly cause Gemini to crash. #12870 |
|
Thank you for the detailed explanation, @lifefloating. Your clarification about using Regarding the choice between options A and B, this is indeed a crucial design decision that impacts user experience and system robustness. As @galz10 mentioned, the ideal scenario is to avoid outright crashes due to incorrect schema, while still providing clear feedback. Let's consider the implications:
Given the context of preventing unpredictable crashes (as highlighted by issue #12870) and providing clear user guidance, a graceful exit with a friendly error message (Option A) often strikes a good balance. It enforces correct configuration while still being user-friendly. However, the ultimate decision depends on the desired product behavior and tolerance for configuration errors in production environments. @galz10, what are your thoughts on these two options, especially considering the balance between strictness, user experience, and preventing unpredictable runtime issues? |
|
I think option a is probably the best approach, i think that's what we sort of do now just in an ungraceful manner. Is option a the one you implemented already ? |
yes Option A is the current implementation method @galz10 |
…or settings schema (google-gemini#12929)
Summary
Fixes CLI crash caused by invalid configuration files by adding runtime validation for
settings.json. Previously, malformed settings would cause the CLI to crash on startup without clear error messages. Now validates settings at startup and provides helpful error messages guiding users to fix configuration issues.Details
Added comprehensive runtime validation for settings configuration:
settings-validation.tsmodule: Dynamically builds Zod validation schemas from the existingsettingsSchemadefinitions, ensuring validation logic stays in sync with schema definitionssettings-validation.test.tswith test cases covering various validation scenariosExample of invalid configuration that now fails gracefully:
{ "model": { "name": { "skipNextSpeakerCheck": true } } }Instead of crashing, users now receive a detailed error message indicating the issue and how to fix it.
Related Issues
Fixes #12870
How to Validate
Test valid configuration:
npm run build # Verify CLI starts normally with valid settings.jsonTest invalid configuration - Create an invalid
settings.json:{ "model": { "name": { "skipNextSpeakerCheck": true } } }# Should display clear error message and exit gracefully (not crash)Run validation tests:
npm test packages/cli/src/config/settings-validation.test.tsVerify error message quality: Check that error messages include:
Pre-Merge Checklist