test(@angular/cli): add CI validation for MCP example code #31174
Conversation
This commit introduces a pre-built, minimal, standalone Angular application that will serve as a validation harness for the code snippets in the `find_examples` tool. This harness, located in `tools/example-validation-harness`, will be used by a new CI script to programmatically compile and type-check all example code. Using a full Angular project environment ensures a high-fidelity validation that covers TypeScript, Angular-specific syntax, and HTML templates.
This commit introduces a new CI check to validate the correctness of code snippets and markdown structure within the `find_examples` markdown files. To support this, the previous the `example_db_generator.js` script has been refactored and consolidated into a single, unified, mode-driven script: `process_examples.mjs`. This improves maintainability and ensures consistent parsing and validation logic. The new script provides two new validation modes for CI: - `validate-structure`: Parses all examples to validate their front matter and ensure they adhere to the required markdown heading structure. - `validate-code`: Uses a pre-built Angular application harness to perform a full `ng build` on all example code, ensuring compilation and template correctness. This provides a strong guarantee of the quality and reliability of the code and structure of all examples served by the `find_examples` tool.
b433d5e to
f1726d0
Compare
| @@ -0,0 +1,21 @@ | |||
| { | |||
There was a problem hiding this comment.
NIT: should this has it's own pnpm lock and workspace file, seeing that there are a seperate pnpm i in the script?
| @@ -59,6 +59,8 @@ jobs: | |||
| uses: angular/dev-infra/github-actions/linting/licenses@5043638fd8529765b375831a4679b9013141b326 | |||
| - name: Check tooling setup | |||
| run: pnpm check-tooling-setup | |||
| - name: Validate MCP Example Markdown | |||
There was a problem hiding this comment.
Nit: consider moving this to before the license checks so that it's always executed even when there are license failures, this is useful for automated dependency bumps. Potentially move Check Package Licenses as the last step
| @@ -0,0 +1,28 @@ | |||
| { | |||
There was a problem hiding this comment.
Instead of creating a application and commit it, should we use the current setup we have to generate an application on the fly? This is similar to other validations and it would make it easier to keep the app in code and deps in sync.
See:
angular-cli/scripts/json-help.mts
Lines 34 to 35 in a195db3
There was a problem hiding this comment.
Thinking more about this, instead of added another CI step, why not simply add a bazel test?
diff --git a/packages/angular/cli/lib/examples/BUILD.bazel b/packages/angular/cli/lib/examples/BUILD.bazel
new file mode 100644
index 000000000..b1f9df0c5
--- /dev/null
+++ b/packages/angular/cli/lib/examples/BUILD.bazel
@@ -0,0 +1,16 @@
+
+filegroup(
+ name = "examples",
+ srcs = glob( ["**/*.md"]),
+)
+
+sh_test(
+ name = "test",
+ srcs = ["//tools:ng_example_db"],
+ data = [":examples", "//tools:ng_example_db"],
+ args = [
+ "$(locations :examples)",
+ "--mode validate-structure",
+ ],
+)
diff --git a/tools/process_examples.mjs b/tools/process_examples.mjs
index 143f8d95a..2939bb9ea 100755
--- a/tools/process_examples.mjs
+++ b/tools/process_examples.mjs
@@ -162,14 +162,12 @@ function parseExampleFile(filePath) {
}
// --- Mode Implementations ---
-
function runValidateStructure(examplesPath) {
console.log('Validating markdown structure of all example files...');
- const exampleFiles = globSync('**/*.md', { cwd: examplesPath });
- for (const file of exampleFiles) {
- parseExampleFile(join(examplesPath, file));
+ for (const file of examplesPath) {
+ parseExampleFile(file);
}
- console.log(`Successfully validated structure of ${exampleFiles.length} files.`);
+ console.log(`Successfully validated structure of ${examplesPath.length} files.`);
}
function runGenerateDb(examplesPath, outputPath) {
@@ -357,7 +355,8 @@ function main() {
},
},
});
- const examplesPath = positionals[0] ?? '.';
+
+ const examplesPath = positionals ?? '.';
const mode = values.mode;
console.log(`Running example processor in '${mode}' mode.`);
@@ -371,7 +370,7 @@ function main() {
runValidateCode(examplesPath);
break;
case 'generate-db':
- runGenerateDb(examplesPath, values.output);
+ runGenerateDb(examplesPath[0], values.output);
break;
default:
throw new Error(
In a follow up we could also update cli_example_db to run the filegroup and runfiles.
IMO: I think it easier in terms of maintainability and future proofing to have all tests run with a single command and under bazel, instead of having multiple commands
| case 'validate-structure': | ||
| runValidateStructure(examplesPath); | ||
| break; | ||
| case 'validate-code': |
There was a problem hiding this comment.
Where is this mode being used?
| console.log(`Successfully generated database with ${parsedExamplesForDb.length} examples.`); | ||
| } | ||
|
|
||
| function runValidateCode(examplesPath) { |
There was a problem hiding this comment.
You we absolutely need to build the code using the CLI? Can't we just do it inside bazel using ng_project
load("@rules_angular//src/ng_project:index.bzl", "ng_project")
package(default_visibility = ["//visibility:public"])
load("//tools:ts_json_schema.bzl", "ts_gen")
ts_extractor(
name = "if-block-sources",
src = "if-block.md",
)
ng_project(
name = "if-block-example",
srcs = [
"if-block.ts",
],
testonly = False,
tsconfig= "//:test-tsconfig",
deps = [
"//:node_modules/@angular/common",
"//:node_modules/@angular/compiler",
"//:node_modules/@angular/compiler-cli",
"//:node_modules/@angular/core",
"//:node_modules/@angular/platform-browser",
"//:node_modules/@angular/router",
],
)ts_extractor should be simply the code in here to extract the TS from md and output a TS.
Happy to help with this if you need.
This commit introduces a new CI check to validate the correctness of code snippets and markdown structure within the
find_examplesmarkdown files.To support this, the previous the
example_db_generator.jsscript has been refactored and consolidated into a single, unified, mode-driven script:process_examples.mjs. This improves maintainability and ensures consistent parsing and validation logic.The new script provides two new validation modes for CI:
validate-structure: Parses all examples to validate their front matter and ensure they adhere to the required markdown heading structure.validate-code: Uses a pre-built Angular application harness to perform a fullng buildon all example code, ensuring compilation and template correctness.NOTE: Only the structure checks are enabled within CI at this time.