feat: add evaluator scoring #101
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed everything up to d461670 in 1 minute and 39 seconds. Click for details.
- Reviewed
409lines of code in4files - Skipped
0files when reviewing. - Skipped posting
3draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. src/client/resources/evaluators.ts:93
- Draft comment:
If both 'traceId' and 'spanId' are provided, the code will only use 'traceId'. Consider explicitly handling or validating this case to avoid silent ambiguity. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% The comment suggests handling a case that's impossible due to TypeScript's type system. The ScoreOptions type explicitly makes traceId and spanId mutually exclusive through its union type definition. TypeScript would catch any attempt to provide both at compile time. The comment shows a misunderstanding of the type system. Could there be runtime scenarios where the type checking is bypassed? Could there be cases where the input comes from external sources? The function takes ScoreOptions as its parameter which is enforced by TypeScript. Even if called from JavaScript, the type guard checks ('traceId' in options) provide runtime safety. The comment should be deleted because it suggests handling an impossible case that's already prevented by the type system and runtime checks.
2. src/client/resources/evaluators.ts:107
- Draft comment:
Add a newline at the end of the file to conform with standard file formatting. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
3. test/evaluators-client.test.ts:223
- Draft comment:
Consider adding tests for the scenario where both 'traceId' and 'spanId' are supplied to ensure the API behaves as expected in that case. - Reason this comment was not posted:
Confidence changes required:50%<= threshold50%None
Workflow ID: wflow_XZI0GHn0hRoxo8Qp
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 1150b4f in 48 seconds. Click for details.
- Reviewed
151lines of code in2files - Skipped
0files when reviewing. - Skipped posting
3draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. src/client/resources/evaluators.ts:4
- Draft comment:
Enum value change from 'SDK' to 'Code' is correctly applied. Ensure any external docs or comments are updated accordingly. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. src/client/resources/evaluators.ts:97
- Draft comment:
Endpoint URL updated from '/v1/evaluators/score' to '/v1/evaluator-scores'. Consider extracting this endpoint into a constant for easier maintenance. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
3. test/evaluators-client.test.ts:38
- Draft comment:
Test adjustments correctly reflect the endpoint and source changes. All assertions now expect 'Code' and new URL. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_UKBzVWPgLCv385Mf
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 527138c in 1 minute and 29 seconds. Click for details.
- Reviewed
202lines of code in4files - Skipped
0files when reviewing. - Skipped posting
4draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. src/cli.ts:41
- Draft comment:
Removed eslint-disable comments for global vars. Ensure declaring globals with 'var' is desired. - Reason this comment was not posted:
Comment looked like it was already resolved.
2. src/client/resources/evaluators.ts:4
- Draft comment:
Added trailing comma in enum member for 'Code' improves diff readability. Verify that this formatting change is intentional. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =40%<= threshold50%The comment is asking the PR author to verify if the formatting change is intentional, which violates the rule against asking for confirmation of intention. However, it also provides a suggestion about improving diff readability, which is useful. The comment could be rephrased to focus on the suggestion rather than asking for verification.
3. src/evaluations.ts:23
- Draft comment:
Removed eslint-disable comments in global declarations. Confirm that using 'var' for globals is acceptable in your codebase. - Reason this comment was not posted:
Comment was on unchanged code.
4. test/evaluators-client.test.ts:52
- Draft comment:
Refactored assertions to use deepStrictEqual for payload verification. This improves test conciseness—ensure no extra keys are unintentionally included. - Reason this comment was not posted:
Comment looked like it was already resolved.
Workflow ID: wflow_1X5Zxdd5YHpyaijN
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 5d42e6d in 1 minute and 1 seconds. Click for details.
- Reviewed
71lines of code in2files - Skipped
0files when reviewing. - Skipped posting
2draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. src/client/resources/evaluators.ts:97
- Draft comment:
Updated endpoint URL. Consider extracting '/v1/evaluators/score' as a constant for reuse. - Reason this comment was not posted:
Confidence changes required:33%<= threshold50%None
2. test/evaluators-client.test.ts:38
- Draft comment:
Test mocks updated to use '/v1/evaluators/score'. Consistent with client change. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_s469vpW8s4m98AX5
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
Adds evaluator scoring feature with
EvaluatorsResourceinLaminarClient, including tests for scoring methods and error handling.EvaluatorsResourceclass insrc/client/resources/evaluators.tsto handle evaluator scoring.score()method inEvaluatorsResourceto create scores usingtraceIdorspanId.traceIdnorspanIdis provided.EvaluatorsResourcetoLaminarClientinsrc/client/index.ts.evaluatorsgetter inLaminarClient.evaluators-client.test.tsto testEvaluatorsResourcemethods.traceIdandspanId, including error handling for missing IDs.package.jsonfrom0.6.14to0.6.15.src/cli.tsandsrc/evaluations.ts.This description was created by
for 5d42e6d. You can customize this summary. It will automatically update as commits are pushed.