fix(tools): change the way how screenshot is cached #387
Conversation
|
@devsargam is attempting to deploy a commit to the Antiwork Team on Vercel. A member of the Team first needs to authorize it. |
|
|
|
Currently, I have added new way of taking screenshot but also supporting the older way of doing so. If we don't want backwards compatibility please let me know! |
There was a problem hiding this comment.
Hey @devsargam, thanks for the contribution. I've added some comments.
| "identifier" in this.testContext.currentTest | ||
| ) { | ||
| const testCase = this.testContext.currentTest as TestCase; | ||
| if (testCase.identifier) { |
There was a problem hiding this comment.
A testCase always has an identifier, generated via transform. The schema should be updated with a pipe to make it strict()
// ...
.transform((data) => {
const hashInput = `${data.name}:${data.filePath}:${JSON.stringify(data.expectations)}`;
// Low collision risk for datasets under 65,000 tests
data.identifier = createHash(hashInput, { length: 8 });
return data;
})
.pipe(z.object({ identifier: z.string() }).strict());There was a problem hiding this comment.
const TestCaseSchema = z
.object({
name: z.string(),
filePath: z.string(),
payload: z.any().optional(),
fn: TestCaseFunctionSchema.optional(),
expectations: z.array(TestCaseExpectationsSchema).default([]),
beforeFn: TestCaseFunctionSchema.optional(),
afterFn: TestCaseFunctionSchema.optional(),
directExecution: z.boolean().optional().default(false),
identifier: z.string().optional(),
})
.strict()
.transform((data) => {
const hashInput = `${data.name}:${data.filePath}:${JSON.stringify(data.expectations)}`;
// Low collision risk for datasets under 65,000 tests
data.identifier = createHash(hashInput, { length: 8 });
return data;
});Here, regardless of whether user passes identifier or not, the original is going to be overwritten by what's there in the transform? Is this intentional?
In preparation for #387 Extract test execution state management into a dedicated `TestRun` class, replacing the previous `TestResult` type. This improves encapsulation of test state transitions (pending -> running -> passed/failed) and token usage tracking. In #387, `TestRun` will be used to set the timestamp when the test runs starts, so that can it can be used to store screenshots in a folder having the same name as the test run cache file.
|
TODOS:
|
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
| }; | ||
|
|
||
| case InternalActionEnum.RUN_CALLBACK: { | ||
| if (!this.testContext?.currentTest) { |
There was a problem hiding this comment.
testContext must always be present.
| this.testContext = newContext; | ||
| } | ||
|
|
||
| private cleanupScreenshots(): void { |
There was a problem hiding this comment.
TestRunRepository deals with cleaning up unwanted runs now.
| const testFileExists = existsSync( | ||
| path.join(process.cwd(), entry.test.filePath), | ||
| ); |
There was a problem hiding this comment.
Rudimentary check if the entry is orphaned.
There was a problem hiding this comment.
Replaced mostly by TestRunRepository
| const testRun = new TestRun(testCase); | ||
| testRun.markRunning(); |
There was a problem hiding this comment.
This is now initialized by the caller of this function.
| }; | ||
|
|
||
| export class TestRunRepository { | ||
| public static VERSION = 2; |
There was a problem hiding this comment.
Increased from 1 previously, and using numbers now. Generally, test runs with previous versions are automatically discarted, their metadata is different.
| (testRun) => | ||
| testRun.status === "passed" && | ||
| testRun.version === TestRunRepository.VERSION && | ||
| !testRun.fromCache, |
There was a problem hiding this comment.
Test runs from cache don't have AI steps, so they are not useful for executing future runs off them.
| } | ||
| // If no passed run, keep only up to MAX_RUNS_PER_TEST most recent runs | ||
| else { | ||
| const MAX_RUNS_PER_TEST = 1; |
There was a problem hiding this comment.
This can be easily changed to keep multiple copies if needed.
| testRun.runId = cacheEntry.metadata.runId; | ||
| testRun.timestamp = cacheEntry.metadata.timestamp; | ||
| testRun.version = | ||
| typeof cacheEntry.metadata.version === "string" |
There was a problem hiding this comment.
AI-generated, haven't reviewed the tests.
|
okay, let me know if I need to make any changes |
Part of #287
The core change is moving from a shared
screenshotsfolder to dedicated folders per test run. This improves organization and makes test results easier to manage and reference.