LCORE-1849: Get the providers e2e tests to run properly#1596
Conversation
WalkthroughE2E test workflows and configs updated to improve connectivity diagnostics, standardize OpenAI env defaults, remove FAISS vector_io, and add transient HTTP retry logic used by test steps; minor workflow timing and image overrides adjusted. Changes
Sequence Diagram(s)sequenceDiagram
participant TestStep as Test Step
participant RetryUtil as request_with_transient_retry
participant RemoteAPI as Remote API (/v1/models)
participant Logger as show_logs
Note right of RetryUtil: rgba(100,150,200,0.5)
TestStep->>RetryUtil: call(method, url, json?, headers, timeout)
RetryUtil->>RemoteAPI: requests.request(...)
alt ConnectionError
RemoteAPI-->>RetryUtil: raises ConnectionError
RetryUtil-->>RetryUtil: sleep(delay) then retry (up to N)
RetryUtil->>RemoteAPI: requests.request(...) (retry)
end
RetryUtil-->>TestStep: Response or raise last ConnectionError
alt Non-200 or curl exit failure (workflows)
TestStep->>Logger: show_logs + exit 1
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
.github/workflows/e2e_tests_rhaiis.yaml (1)
35-36: 🧹 Nitpick | 🔵 TrivialInconsistency: bypass matrix value for env var.
Lines 23/35 set
e2e_default_model: ${{ vars.RHAIIS_MODEL }}andE2E_DEFAULT_MODEL_OVERRIDE: ${{ vars.RHAIIS_MODEL }}independently.e2e_tests_rhelai.yaml(Line 29) reads from${{ matrix.e2e_default_model }}for the override, which is the cleaner pattern. Consider aligning so the matrix definition is the single source of truth:🔧 Suggested change
- E2E_DEFAULT_MODEL_OVERRIDE: ${{ vars.RHAIIS_MODEL }} - E2E_DEFAULT_PROVIDER_OVERRIDE: vllm + E2E_DEFAULT_MODEL_OVERRIDE: ${{ matrix.e2e_default_model }} + E2E_DEFAULT_PROVIDER_OVERRIDE: ${{ matrix.e2e_default_provider }}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/e2e_tests_rhaiis.yaml around lines 35 - 36, The env var override is duplicated: E2E_DEFAULT_MODEL_OVERRIDE is set directly while the workflow also defines matrix.e2e_default_model; change the workflow to use the matrix value as the single source of truth by wiring E2E_DEFAULT_MODEL_OVERRIDE to matrix.e2e_default_model (instead of vars.RHAIIS_MODEL) so that the job reads the model from matrix.e2e_default_model consistently with e2e_tests_rhelai.yaml; update any similar direct assignments (e.g., E2E_DEFAULT_PROVIDER_OVERRIDE) to reference their matrix counterparts if present to keep the matrix as the single source of truth.tests/e2e/features/steps/common_http.py (1)
169-206: 🧹 Nitpick | 🔵 TrivialInconsistency: non-REST helpers still use raw
requests.
access_non_rest_api_endpoint_get(Line 180) andaccess_non_rest_api_endpoint_post(Line 204) still callrequests.get/requests.postdirectly and won't benefit from the new transient-retry behavior. If the rationale forrequest_with_transient_retryis "container restarts in CI", these/v1/liveness-style probes are arguably the most likely to hit a freshly-restarted service. Consider routing them through the same wrapper for consistency.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/e2e/features/steps/common_http.py` around lines 169 - 206, Both helpers access_non_rest_api_endpoint_get and access_non_rest_api_endpoint_post call requests.get/requests.post directly; change them to call the existing request_with_transient_retry wrapper so these probes get transient-retry behavior. For access_non_rest_api_endpoint_get replace requests.get(url, timeout=DEFAULT_TIMEOUT) with a call to request_with_transient_retry("GET", url, headers=maybe-auth-headers, timeout=DEFAULT_TIMEOUT) (add headers from context.auth_headers if present), and for access_non_rest_api_endpoint_post replace requests.post(... json=data ...) with request_with_transient_retry("POST", url, json=data, headers=context.auth_headers if present, timeout=DEFAULT_TIMEOUT); keep the existing assertions (e.g., context.text) and continue to assign the returned response to context.response.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/e2e_tests_providers.yaml:
- Around line 312-314: The inline comment for the step named "Wait for watsonx
library mode to finish" is incorrect: the run uses "sleep 3600" (60 minutes) but
the comment says "120 minutes"; update the comment to reflect 60 minutes (e.g.,
change "# 120 minutes" to "# 60 minutes" or "# 3600 seconds / 60 minutes") so
the comment matches the actual command in that step.
In @.github/workflows/e2e_tests_rhelai.yaml:
- Around line 183-191: The current check only inspects the curl exit code
(CURL_EC) and can miss HTTP 4xx/5xx responses; update the curl call that
populates MODEL_JSON to also capture the HTTP status (use curl -w "%{http_code}"
or similar) into a separate variable (e.g., HTTP_STATUS) or append it so you can
separate the body and status, then assert that HTTP_STATUS equals 200 before
running the jq validation that looks for the LLM; if status is not 200, call
show_logs and exit with a clear message including the HTTP status and the
MODEL_JSON body for debugging (referencing the MODEL_JSON, CURL_EC and show_logs
symbols in the diff).
- Around line 132-142: The connectivity step currently swallows curl errors by
using "|| printf '\n000'" so RESP/HTTP_CODE only shows "000" without curl's exit
status; modify the run block that builds RESP/BODY/HTTP_CODE to capture curl's
exit code (e.g., save $? immediately after curl into CURL_EXIT), and when
CURL_EXIT is non-zero print a clear diagnostic including CURL_EXIT plus the raw
curl stderr/output (alongside existing HTTP_CODE and BODY); apply the same
change to the corresponding step in the e2e_tests_rhaiis.yaml so both curl calls
(the RESP/HTTP_CODE logic) emit curl's exit code and any stderr when
connectivity fails.
- Around line 4-5: The workflow trigger is too broad — change the top-level
trigger in e2e_tests_rhelai.yaml (and mirror the same change in
e2e_tests_rhaiis.yaml and e2e_tests_providers.yaml) from a bare "on: push" to a
scoped trigger such as "on: { push: { branches: [main] }, pull_request_target: {
branches: [main] }, workflow_dispatch: {} }" (or include any additional allowed
branches like release/*) so E2E runs only on intended branches and still allows
manual runs via workflow_dispatch; update the trigger block to match the pattern
used in e2e_tests.yaml.
In `@tests/e2e/configs/run-bedrock.yaml`:
- Around line 154-158: Remove the stale top-level vector_stores block that
references default_provider_id: faiss and the default_embedding_model
(provider_id: sentence-transformers, model_id: all-mpnet-base-v2) from
tests/e2e/configs/run-bedrock.yaml; also make the same removal in
tests/e2e/configs/run-watsonx.yaml to avoid an orphaned configuration now that
providers.vector_io is empty. Locate the block by the key vector_stores and
delete the entire mapping (default_provider_id and default_embedding_model) so
no FAISS reference remains.
In `@tests/e2e/configs/run-vertexai.yaml`:
- Line 67: The config sets vector_io: [] but still references a faiss provider
via vector_stores.default_provider_id and registers the builtin::rag tool group,
causing runtime failures; either re-add an inline vector_io provider (e.g.,
inline::faiss) and the corresponding vector_stores entry including
default_embedding_model, or remove RAG entirely by deleting vector_io from apis,
removing the builtin::rag tool group registration, and clearing/omitting
vector_stores.default_provider_id and default_embedding_model; apply the same
fix consistently across the other e2e configs (run-azure.yaml, run-bedrock.yaml,
run-ci.yaml, run-rhaiis.yaml, run-rhelai.yaml, run-watsonx.yaml).
In `@tests/e2e/features/steps/common_http.py`:
- Around line 225-236: The current branching in the HTTP helper uses the HTTP
verb check (method in ("GET", "DELETE")) which misses other bodyless verbs and
causes the assert on context.text in functions like request_with_transient_retry
to fail; change the logic to decide whether to pass a JSON body based on
presence of context.text (e.g., if context.text is None then call
request_with_transient_retry without json=, otherwise
json=json.loads(context.text)), keeping the same headers, timeout
(DEFAULT_TIMEOUT) and existing transient-retry behavior so HEAD/OPTIONS or
intentionally bodyless PUT/PATCH requests are handled correctly.
In `@tests/e2e/utils/utils.py`:
- Around line 68-92: The function request_with_transient_retry currently retries
on any requests.exceptions.ConnectionError for all HTTP methods which can replay
non-idempotent requests (e.g., POST) and cause duplicate backend work; update
request_with_transient_retry to only retry for safe/idempotent methods (at
minimum allow retries for "GET", "HEAD", "DELETE") by checking
kwargs.get("method", "GET").upper() before the retry loop (use
E2E_HTTP_TRANSIENT_MAX_ATTEMPTS and E2E_HTTP_TRANSIENT_DELAY_S as before), or if
you prefer keep current behavior add a clear docstring note warning about
duplicate side-effects for non-idempotent methods; also narrow the last_err
annotation to requests.exceptions.ConnectionError | None to reflect the actual
captured exception type.
- Around line 84-90: Update the transient HTTP retry loop that calls
requests.request(**kwargs) (the loop using E2E_HTTP_TRANSIENT_MAX_ATTEMPTS and
E2E_HTTP_TRANSIENT_DELAY_S) to use linear/exponential backoff (e.g., sleep
E2E_HTTP_TRANSIENT_DELAY_S * (attempt + 1) or multiply by 2**attempt) and log
each retry with the attempt index and exception (use the module test logger or
requests logger) so retries are visible; also add a final error log before
re-raising or returning the last exception if all attempts fail. Ensure the
changes reference the same retry loop variables
(E2E_HTTP_TRANSIENT_MAX_ATTEMPTS, E2E_HTTP_TRANSIENT_DELAY_S, last_err) and
preserve raising/return behavior.
---
Outside diff comments:
In @.github/workflows/e2e_tests_rhaiis.yaml:
- Around line 35-36: The env var override is duplicated:
E2E_DEFAULT_MODEL_OVERRIDE is set directly while the workflow also defines
matrix.e2e_default_model; change the workflow to use the matrix value as the
single source of truth by wiring E2E_DEFAULT_MODEL_OVERRIDE to
matrix.e2e_default_model (instead of vars.RHAIIS_MODEL) so that the job reads
the model from matrix.e2e_default_model consistently with e2e_tests_rhelai.yaml;
update any similar direct assignments (e.g., E2E_DEFAULT_PROVIDER_OVERRIDE) to
reference their matrix counterparts if present to keep the matrix as the single
source of truth.
In `@tests/e2e/features/steps/common_http.py`:
- Around line 169-206: Both helpers access_non_rest_api_endpoint_get and
access_non_rest_api_endpoint_post call requests.get/requests.post directly;
change them to call the existing request_with_transient_retry wrapper so these
probes get transient-retry behavior. For access_non_rest_api_endpoint_get
replace requests.get(url, timeout=DEFAULT_TIMEOUT) with a call to
request_with_transient_retry("GET", url, headers=maybe-auth-headers,
timeout=DEFAULT_TIMEOUT) (add headers from context.auth_headers if present), and
for access_non_rest_api_endpoint_post replace requests.post(... json=data ...)
with request_with_transient_retry("POST", url, json=data,
headers=context.auth_headers if present, timeout=DEFAULT_TIMEOUT); keep the
existing assertions (e.g., context.text) and continue to assign the returned
response to context.response.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: a3561b0c-7a36-4777-aaea-150cb628eabc
📒 Files selected for processing (12)
.github/workflows/e2e_tests_providers.yaml.github/workflows/e2e_tests_rhaiis.yaml.github/workflows/e2e_tests_rhelai.yamltests/e2e/configs/run-azure.yamltests/e2e/configs/run-bedrock.yamltests/e2e/configs/run-rhaiis.yamltests/e2e/configs/run-rhelai.yamltests/e2e/configs/run-vertexai.yamltests/e2e/configs/run-watsonx.yamltests/e2e/features/steps/common_http.pytests/e2e/features/steps/llm_query_response.pytests/e2e/utils/utils.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: Pylinter
- GitHub Check: build-pr
- GitHub Check: unit_tests (3.12)
- GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-on-pull-request
- GitHub Check: E2E: library mode / ci / group 2
- GitHub Check: E2E: library mode / ci / group 1
- GitHub Check: E2E: server mode / ci / group 3
- GitHub Check: E2E Tests for Lightspeed Evaluation job
- GitHub Check: E2E: server mode / ci / group 1
- GitHub Check: E2E: server mode / ci / group 2
- GitHub Check: E2E: library mode / ci / group 3
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Use absolute imports for internal modules:from authentication import get_auth_dependency
Import FastAPI dependencies with:from fastapi import APIRouter, HTTPException, Request, status, Depends
Import Llama Stack client with:from llama_stack_client import AsyncLlamaStackClient
Checkconstants.pyfor shared constants before defining new ones
All modules start with descriptive docstrings explaining purpose
Uselogger = get_logger(__name__)fromlog.pyfor module logging
Type aliases defined at module level for clarity
Use Final[type] as type hint for all constants
All functions require docstrings with brief descriptions
Complete type annotations for parameters and return types in functions
Usetyping_extensions.Selffor model validators in Pydantic models
Use modern union type syntaxstr | intinstead ofUnion[str, int]
UseOptional[Type]for optional type hints
Use snake_case with descriptive, action-oriented function names (get_, validate_, check_)
Avoid in-place parameter modification anti-patterns; return new data structures instead
Useasync deffor I/O operations and external API calls
HandleAPIConnectionErrorfrom Llama Stack in error handling
Use standard log levels with clear purposes: debug, info, warning, error
All classes require descriptive docstrings explaining purpose
Use PascalCase for class names with standard suffixes: Configuration, Error/Exception, Resolver, Interface
Use ABC for abstract base classes with@abstractmethoddecorators
Use@model_validatorand@field_validatorfor Pydantic model validation
Complete type annotations for all class attributes; use specific types, notAny
Follow Google Python docstring conventions with Parameters, Returns, Raises, and Attributes sections
Files:
tests/e2e/utils/utils.pytests/e2e/features/steps/common_http.pytests/e2e/features/steps/llm_query_response.py
tests/e2e/**/*.{py,feature}
📄 CodeRabbit inference engine (AGENTS.md)
Use behave (BDD) framework for end-to-end testing
Files:
tests/e2e/utils/utils.pytests/e2e/features/steps/common_http.pytests/e2e/features/steps/llm_query_response.py
🧠 Learnings (5)
📚 Learning: 2025-09-02T11:14:17.117Z
Learnt from: radofuchs
Repo: lightspeed-core/lightspeed-stack PR: 485
File: tests/e2e/features/steps/common_http.py:244-255
Timestamp: 2025-09-02T11:14:17.117Z
Learning: The POST step in tests/e2e/features/steps/common_http.py (`access_rest_api_endpoint_post`) is intentionally designed as a general-purpose HTTP POST method, not specifically for REST API endpoints, so it should not include context.api_prefix in the URL construction.
Applied to files:
tests/e2e/features/steps/common_http.py
📚 Learning: 2026-04-07T09:20:26.590Z
Learnt from: radofuchs
Repo: lightspeed-core/lightspeed-stack PR: 1467
File: tests/e2e/features/steps/common.py:36-49
Timestamp: 2026-04-07T09:20:26.590Z
Learning: For Behave-based Python tests, rely on Behave’s Context layered stack for attribute lifecycle: Behave pushes a new Context layer when entering feature scope (before_feature) and again for scenario scope (before_scenario). Attributes assigned inside given/when/then steps live on the current scenario layer and are automatically removed when the scenario ends. As a result, step-set attributes should not be expected to persist across scenarios or features, and manual cleanup in after_scenario/after_feature is generally unnecessary for attributes set in step functions. Only perform manual cleanup for attributes that you set explicitly in before_feature/before_scenario, since those live on the respective feature/scenario layers.
Applied to files:
tests/e2e/features/steps/common_http.pytests/e2e/features/steps/llm_query_response.py
📚 Learning: 2026-04-13T13:39:54.963Z
Learnt from: radofuchs
Repo: lightspeed-core/lightspeed-stack PR: 1490
File: tests/e2e/features/environment.py:206-211
Timestamp: 2026-04-13T13:39:54.963Z
Learning: In lightspeed-stack E2E tests under tests/e2e/features, it is intentional to set context.feature_config inside Background/step functions (scenario-scoped Behave layer). The environment.py after_scenario restore logic should only restore configuration when context.scenario_lightspeed_override_active is True; this flag is set by configure_service only when a real config switch occurs (so restore does not run for scenarios without a switch). Additionally, steps/common.py’s module-level _active_lightspeed_stack_config_basename is used to prevent re-applying the same config across subsequent scenarios, ensuring scenario_lightspeed_override_active stays False after the first apply. Therefore, reviewers should not “fix” this flow as if feature_config were incorrectly scoped or if after_scenario restoration is missing—config switching and restoration are meant to happen exactly once per actual switch, not redundantly per scenario.
Applied to files:
tests/e2e/features/steps/common_http.pytests/e2e/features/steps/llm_query_response.py
📚 Learning: 2025-12-18T10:21:09.038Z
Learnt from: are-ces
Repo: lightspeed-core/lightspeed-stack PR: 935
File: run.yaml:114-115
Timestamp: 2025-12-18T10:21:09.038Z
Learning: In Llama Stack version 0.3.x, telemetry provider configuration is not supported under the `providers` section in run.yaml configuration files. Telemetry can be enabled with just `telemetry.enabled: true` without requiring an explicit provider block.
Applied to files:
tests/e2e/configs/run-bedrock.yamltests/e2e/configs/run-rhaiis.yamltests/e2e/configs/run-azure.yamltests/e2e/configs/run-rhelai.yaml
📚 Learning: 2026-03-02T16:38:30.287Z
Learnt from: CR
Repo: lightspeed-core/lightspeed-stack PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-03-02T16:38:30.287Z
Learning: Applies to AGENTS.md : Document agent implementations and their configurations in AGENTS.md
Applied to files:
tests/e2e/configs/run-bedrock.yamltests/e2e/configs/run-watsonx.yaml
🔇 Additional comments (17)
.github/workflows/e2e_tests_rhaiis.yaml (2)
4-5: Sameon: pushconcern ase2e_tests_rhelai.yaml.See the comment on
.github/workflows/e2e_tests_rhelai.yamlLine 4-5; consider scoping the trigger here as well.
189-197: Same connectivity-check concerns ase2e_tests_rhelai.yaml.Capturing only curl exit code masks HTTP 4xx/5xx; see the comment on
.github/workflows/e2e_tests_rhelai.yamlLines 183-191 for the suggested fix..github/workflows/e2e_tests_providers.yaml (3)
4-5: Sameon: pushconcern as other provider workflows.See the comment on
.github/workflows/e2e_tests_rhelai.yamlLines 4-5 for the recommended trigger scoping..github/workflows/e2e_tests.yamlalready uses[push, pull_request_target], so aligning these workflows would be consistent.
288-296: Same curl-exit-only handling as other provider workflows.See the comment on
.github/workflows/e2e_tests_rhelai.yamlLines 183-191 for context.
316-318:⚠️ Potential issue | 🟠 MajorAdd explanation and verify consistency across connectivity test and test execution.
The prefix-stripping step is necessary due to a naming convention mismatch: the connectivity test expects and confirms the prefixed form (
watsonx/meta-llama/llama-3-3-70b-instruct), but the Behave test framework requires the unprefixed model name forcontext.default_model. The override step (line 317) strips the prefix, leaving onlymeta-llama/llama-3-3-70b-instructfor test execution.However, this inconsistency is not documented. Add an inline comment explaining why the override is needed (llama-stack vs Behave naming conventions), and verify that test steps using
context.default_modeldon't attempt to validate against the fullprovider_resource_idformat, which would include the prefix again.⛔ Skipped due to learnings
Learnt from: asimurka Repo: lightspeed-core/lightspeed-stack PR: 1198 File: src/utils/query.py:91-110 Timestamp: 2026-02-23T14:07:37.612Z Learning: In `src/utils/query.py`, the `validate_model_provider_override` function should only flag an override when: (1) provider is explicitly specified (separated format), or (2) model contains "/" (concatenated provider/model format). Specifying just a model name without provider is not considered an override and uses the default provider.Learnt from: radofuchs Repo: lightspeed-core/lightspeed-stack PR: 1490 File: tests/e2e/features/environment.py:206-211 Timestamp: 2026-04-13T13:39:59.316Z Learning: In lightspeed-stack e2e tests (tests/e2e/features/), `context.feature_config` is intentionally set inside Background/step functions (scenario-scoped Behave layer). The `after_scenario` restore logic in `environment.py` only restores config when `context.scenario_lightspeed_override_active` is True, which is only set by `configure_service` when an actual config switch occurs. The module-level `_active_lightspeed_stack_config_basename` in `tests/e2e/features/steps/common.py` prevents re-applying the same config in subsequent scenarios (making `scenario_lightspeed_override_active` stay False). This means the ephemeral nature of step-set context attributes is intentional — the design ensures config restore happens exactly once per actual switch, not redundantly on every scenario.Learnt from: radofuchs Repo: lightspeed-core/lightspeed-stack PR: 485 File: tests/e2e/features/steps/common_http.py:244-255 Timestamp: 2025-09-02T11:14:17.117Z Learning: The POST step in tests/e2e/features/steps/common_http.py (`access_rest_api_endpoint_post`) is intentionally designed as a general-purpose HTTP POST method, not specifically for REST API endpoints, so it should not include context.api_prefix in the URL construction.tests/e2e/configs/run-watsonx.yaml (2)
152-158: Same FAISS-stale / safety-mismatch concerns asrun-bedrock.yaml.
vector_stores.default_provider_id: faissremains whileproviders.vector_iois empty (Line 68), andshields[].provider_shield_idisopenai/gpt-4o-miniwhile the openai inference provider declaresallowed_models: ["gpt-4o-mini"](Line 33). See the comments ontests/e2e/configs/run-bedrock.yamlLines 144-146 and 154-158 for the recommended fixes.
2-2: No action needed. Theimage_name: watsonx-configurationfollows the established pattern in the codebase where provider-specific e2e configs use provider-scoped image names (e.g.,azure-configuration,vertexai-configuration,rhaiis-configuration,rhelai-configuration). Generic/CI configs usestarter. This naming is intentional and consistent.> Likely an incorrect or invalid review comment.tests/e2e/features/steps/llm_query_response.py (2)
110-112: LGTM — direct call replaced with the shared retry wrapper, signature/timeout preserved.The non-idempotency concern for retried POSTs is already noted on the helper definition in
tests/e2e/utils/utils.py, no need to repeat per call site.
142-161: Streaming + retry: confirm intended behavior on mid-stream resets.
request_with_transient_retryonly retries onrequests.exceptions.ConnectionErrorraised during the initialrequests.request(**kwargs)call. Oncestream=Truereturns a Response, the actual SSE body is consumed in_read_streamed_response(resp)(Line 151), where mid-streamChunkedEncodingErroris already swallowed (_read_streamed_response). That's reasonable for the "server may close stream after sending an error event" case, but it does mean a transient connection reset occurring after headers are sent will not retry — only connection setup is retried. If that's the intended scope (and given the rationale in the wrapper docstring, it appears to be), this is fine; just calling out so the limitation is explicit.tests/e2e/configs/run-bedrock.yaml (1)
144-146: No issue. The configuration is correct.The shield's
provider_shield_id: openai/gpt-4o-minimatches the expected model ID format. When the openai provider registers its models, they are exposed via Llama Stack'smodels.list()with theopenai/prefix in theiridfield (e.g.,openai/gpt-4o-mini). The shield validation insrc/utils/shields.pychecks whetherprovider_resource_idexists in the available models set, and this check succeeds because the prefixed format is consistent throughout the registration and lookup process. The E2E test suite confirms this configuration works correctly across all test configurations.tests/e2e/configs/run-azure.yaml (2)
57-60: Same literal'********'Braintrust key issue asrun-vertexai.yaml.See the consolidated comment on
run-vertexai.yamllines 56-60 —openai_api_key: '********'is a literal value, not an env reference. Apply the same fix (${env.OPENAI_API_KEY}) or remove theinline::braintrustentry.
68-68: Same dangling-RAG inconsistency asrun-vertexai.yaml.
providers.vector_io: []here, butapis(line 14) still includesvector_io,tool_groups(line 154) still registersbuiltin::rag, andvector_stores.default_provider_id: faiss(line 157) still points at the removed provider. See the consolidated comment onrun-vertexai.yamlline 67 — apply the same all-or-nothing cleanup.tests/e2e/configs/run-rhaiis.yaml (2)
57-60: Same literal'********'Braintrust key issue.Same as
run-vertexai.yamllines 56-60 — apply the same fix or drop the provider.
68-68: Same dangling-RAG inconsistency.Same as
run-vertexai.yamlline 67 —vector_io: []is inconsistent with the still-presentapis: [... vector_io],builtin::ragtool group (line 154), andvector_stores.default_provider_id: faiss(line 157). Pick one direction (restore faiss provider, or remove all RAG wiring) and apply consistently.tests/e2e/configs/run-rhelai.yaml (2)
57-60: Same literal'********'Braintrust key issue.Same as
run-vertexai.yamllines 56-60 — apply the same fix or drop the provider.
68-68: Same dangling-RAG inconsistency.Same as
run-vertexai.yamlline 67. Additionally worth confirming:apis(line 14) still containsvector_io;tool_groups(line 154) still registersbuiltin::rag; andvector_stores.default_provider_id: faiss(line 157) still references the removed provider. Apply the all-or-nothing cleanup uniformly.tests/e2e/configs/run-vertexai.yaml (1)
56-60:⚠️ Potential issue | 🟠 MajorLiteral
'********'will be used as the Braintrust API key.
openai_api_key: '********'is the masked placeholder that llama-stack emits when dumping configs (it isn't env-var interpolation). As written, theinline::braintrustprovider will be initialized with the literal string********and any scoring call routed to it will fail (or, worse, leak that string into outbound headers). The same pattern is duplicated inrun-azure.yaml,run-rhaiis.yaml, andrun-rhelai.yaml.If Braintrust scoring is actually exercised by these e2e tests, wire it to the env var; otherwise drop the provider entry to avoid failing at provider init.
🛡️ Proposed fix
- provider_id: braintrust provider_type: inline::braintrust config: - openai_api_key: '********' + openai_api_key: ${env.OPENAI_API_KEY}llama-stack inline::braintrust scoring provider openai_api_key configuration⛔ Skipped due to learnings
Learnt from: radofuchs Repo: lightspeed-core/lightspeed-stack PR: 1490 File: tests/e2e/features/environment.py:377-381 Timestamp: 2026-04-13T13:34:51.052Z Learning: In `tests/e2e/features/environment.py` (lightspeed-core/lightspeed-stack), the hardcoded token `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva` is intentionally a dummy/truncated JWT (the canonical jwt.io documentation example, missing its signature segment). It is not a real credential and does not need to be replaced or loaded from an environment variable. Only a scanner-suppression comment may be warranted if CI noise becomes an issue.
| - name: Wait for watsonx library mode to finish | ||
| if: matrix.environment == 'watsonx' && matrix.mode == 'server' | ||
| run: sleep 7200 # 120 minutes | ||
| run: sleep 3600 # 120 minutes |
There was a problem hiding this comment.
Comment/value mismatch on watsonx wait.
sleep 3600 waits 60 minutes, not the "120 minutes" stated in the comment. The PR summary also mentions reducing this from 7200s to 3600s, so the value is intentional but the comment is stale.
🔧 Suggested fix
- run: sleep 3600 # 120 minutes
+ run: sleep 3600 # 60 minutes📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Wait for watsonx library mode to finish | |
| if: matrix.environment == 'watsonx' && matrix.mode == 'server' | |
| run: sleep 7200 # 120 minutes | |
| run: sleep 3600 # 120 minutes | |
| - name: Wait for watsonx library mode to finish | |
| if: matrix.environment == 'watsonx' && matrix.mode == 'server' | |
| run: sleep 3600 # 60 minutes |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e_tests_providers.yaml around lines 312 - 314, The
inline comment for the step named "Wait for watsonx library mode to finish" is
incorrect: the run uses "sleep 3600" (60 minutes) but the comment says "120
minutes"; update the comment to reflect 60 minutes (e.g., change "# 120 minutes"
to "# 60 minutes" or "# 3600 seconds / 60 minutes") so the comment matches the
actual command in that step.
| run: | | ||
| echo $RHEL_AI_MODEL | ||
| curl -f ${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models -H "Authorization: Bearer ${RHEL_AI_API_KEY}" | ||
| echo "GET ${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" | ||
| RESP=$(curl -sS -w "\n%{http_code}" "${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" \ | ||
| -H "Authorization: Bearer ${RHEL_AI_API_KEY}" || printf '\n000') | ||
| HTTP_CODE=$(echo "$RESP" | tail -n1) | ||
| BODY=$(echo "$RESP" | sed '$d') | ||
| echo "HTTP $HTTP_CODE" | ||
| echo "Response body:" | ||
| echo "$BODY" | ||
| [ "$HTTP_CODE" = "200" ] |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Connectivity step swallows curl failures silently into HTTP code 000.
|| printf '\n000' on Line 136 makes the curl pipeline never fail; only the final [ "$HTTP_CODE" = "200" ] reports failure. That's intentional, but any underlying curl error (DNS, TLS, network) is now reduced to "HTTP 000" with no diagnostic. Consider also capturing curl's exit code (like the Quick connectivity test does) and printing it on failure so debugging RHEL AI / RHAIIS connectivity issues doesn't require re-running with -v.
🔧 Suggested diagnostic improvement
echo "GET ${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models"
- RESP=$(curl -sS -w "\n%{http_code}" "${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" \
- -H "Authorization: Bearer ${RHEL_AI_API_KEY}" || printf '\n000')
+ CURL_EC=0
+ RESP=$(curl -sS -w "\n%{http_code}" "${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" \
+ -H "Authorization: Bearer ${RHEL_AI_API_KEY}") || CURL_EC=$?
HTTP_CODE=$(echo "$RESP" | tail -n1)
BODY=$(echo "$RESP" | sed '$d')
- echo "HTTP $HTTP_CODE"
+ echo "HTTP ${HTTP_CODE:-000} (curl exit ${CURL_EC})"
echo "Response body:"
echo "$BODY"
[ "$HTTP_CODE" = "200" ]Same pattern applies to the corresponding step in .github/workflows/e2e_tests_rhaiis.yaml (Lines 119-128).
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| run: | | |
| echo $RHEL_AI_MODEL | |
| curl -f ${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models -H "Authorization: Bearer ${RHEL_AI_API_KEY}" | |
| echo "GET ${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" | |
| RESP=$(curl -sS -w "\n%{http_code}" "${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" \ | |
| -H "Authorization: Bearer ${RHEL_AI_API_KEY}" || printf '\n000') | |
| HTTP_CODE=$(echo "$RESP" | tail -n1) | |
| BODY=$(echo "$RESP" | sed '$d') | |
| echo "HTTP $HTTP_CODE" | |
| echo "Response body:" | |
| echo "$BODY" | |
| [ "$HTTP_CODE" = "200" ] | |
| run: | | |
| echo $RHEL_AI_MODEL | |
| echo "GET ${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" | |
| CURL_EC=0 | |
| RESP=$(curl -sS -w "\n%{http_code}" "${RHEL_AI_URL}:${RHEL_AI_PORT}/v1/models" \ | |
| -H "Authorization: Bearer ${RHEL_AI_API_KEY}") || CURL_EC=$? | |
| HTTP_CODE=$(echo "$RESP" | tail -n1) | |
| BODY=$(echo "$RESP" | sed '$d') | |
| echo "HTTP ${HTTP_CODE:-000} (curl exit ${CURL_EC})" | |
| echo "Response body:" | |
| echo "$BODY" | |
| [ "$HTTP_CODE" = "200" ] |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e_tests_rhelai.yaml around lines 132 - 142, The
connectivity step currently swallows curl errors by using "|| printf '\n000'" so
RESP/HTTP_CODE only shows "000" without curl's exit status; modify the run block
that builds RESP/BODY/HTTP_CODE to capture curl's exit code (e.g., save $?
immediately after curl into CURL_EXIT), and when CURL_EXIT is non-zero print a
clear diagnostic including CURL_EXIT plus the raw curl stderr/output (alongside
existing HTTP_CODE and BODY); apply the same change to the corresponding step in
the e2e_tests_rhaiis.yaml so both curl calls (the RESP/HTTP_CODE logic) emit
curl's exit code and any stderr when connectivity fails.
| CURL_EC=0 | ||
| MODEL_JSON=$(curl -sS "http://localhost:8080/v1/models?model_type=llm") || CURL_EC=$? | ||
| echo "GET /v1/models?model_type=llm response:" | ||
| echo "$MODEL_JSON" | ||
| if [ "$CURL_EC" -ne 0 ]; then | ||
| echo "❌ GET /v1/models?model_type=llm failed (curl exit $CURL_EC)" | ||
| show_logs | ||
| exit 1 | ||
| } | ||
| fi |
There was a problem hiding this comment.
Capturing only curl exit ignores HTTP 4xx/5xx responses.
The previous curl -sfS short-circuited on any non-2xx HTTP status. The new logic only fails on a non-zero curl exit (network/DNS errors), so a 401/403/500 from /v1/models?model_type=llm would proceed into the jq check and produce a confusing "Expected LLM not listed" failure instead of the actual upstream error code. Consider also capturing the HTTP status (-w "\n%{http_code}") and asserting 200 before the jq validation.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e_tests_rhelai.yaml around lines 183 - 191, The current
check only inspects the curl exit code (CURL_EC) and can miss HTTP 4xx/5xx
responses; update the curl call that populates MODEL_JSON to also capture the
HTTP status (use curl -w "%{http_code}" or similar) into a separate variable
(e.g., HTTP_STATUS) or append it so you can separate the body and status, then
assert that HTTP_STATUS equals 200 before running the jq validation that looks
for the LLM; if status is not 200, call show_logs and exit with a clear message
including the HTTP status and the MODEL_JSON body for debugging (referencing the
MODEL_JSON, CURL_EC and show_logs symbols in the diff).
| vector_stores: | ||
| default_provider_id: faiss | ||
| default_embedding_model: | ||
| default_embedding_model: # Define the default embedding model for RAG | ||
| provider_id: sentence-transformers | ||
| model_id: all-mpnet-base-v2 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect how `vector_stores.default_provider_id` is consumed and whether an empty
# providers.vector_io combined with default_provider_id: faiss is tolerated.
rg -nP -C3 'default_provider_id|vector_stores' --type=py
rg -nP -C3 'default_provider_id' tests/e2e/configsRepository: lightspeed-core/lightspeed-stack
Length of output: 50389
🏁 Script executed:
# Check the bedrock config file
cat -n tests/e2e/configs/run-bedrock.yaml | sed -n '150,165p'Repository: lightspeed-core/lightspeed-stack
Length of output: 497
🏁 Script executed:
# Check the watsonx config file too
cat -n tests/e2e/configs/run-watsonx.yaml | sed -n '150,165p'Repository: lightspeed-core/lightspeed-stack
Length of output: 451
🏁 Script executed:
# Search for how default_provider_id is validated or used in code
rg -nA5 -B2 'default_provider_id' --type=py -g '!test' | head -80Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Search for where default_provider_id is actually consumed or validated
rg -nA3 'default_provider_id|vector_stores' --type=py -g '!test' | head -100Repository: lightspeed-core/lightspeed-stack
Length of output: 6862
🏁 Script executed:
# Look at the run-bedrock.yaml file more comprehensively to understand the full context
cat -n tests/e2e/configs/run-bedrock.yaml | grep -A2 -B2 'vector_io\|vector_stores'Repository: lightspeed-core/lightspeed-stack
Length of output: 731
🏁 Script executed:
# Check if there are any vector_io providers defined in bedrock config
rg -n 'vector_io' tests/e2e/configs/run-bedrock.yamlRepository: lightspeed-core/lightspeed-stack
Length of output: 110
🏁 Script executed:
# Look for usage of vector_stores configuration in the codebase
rg -n 'vector_stores' tests/unit/test_llama_stack_configuration.py | head -30Repository: lightspeed-core/lightspeed-stack
Length of output: 2110
🏁 Script executed:
# Check if there's any schema or validation that handles top-level vector_stores
fd -e py -x rg -l 'vector_stores' {} \; | grep -v test | head -10Repository: lightspeed-core/lightspeed-stack
Length of output: 287
🏁 Script executed:
# Look at the actual config schema or how vector_stores is loaded/validated
rg -n -A5 'vector_stores' llama_stack --type=py | head -80Repository: lightspeed-core/lightspeed-stack
Length of output: 128
🏁 Script executed:
# Look at the config schema
rg -n -A5 'vector_stores' src/models/config.py | head -100Repository: lightspeed-core/lightspeed-stack
Length of output: 370
🏁 Script executed:
# Check llama_stack_configuration.py for vector_stores handling
rg -n -A10 'vector_stores' src/llama_stack_configuration.py | head -80Repository: lightspeed-core/lightspeed-stack
Length of output: 3507
🏁 Script executed:
# Search for where vector_stores default section is used or validated
rg -n 'default_provider_id.*faiss|faiss.*default_provider_id' src --type=pyRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Look for where the top-level vector_stores section is parsed/used
rg -n 'vector_stores' src/models/config.py -A2 -B2 | head -100Repository: lightspeed-core/lightspeed-stack
Length of output: 268
🏁 Script executed:
# Check if there's any validation that checks if default_provider_id exists in vector_io
rg -n 'default_provider_id' src --type=py -B3 -A3 | head -100Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Search for where the top-level vector_stores config (with default_provider_id) is consumed
rg -n 'default_provider_id' src --type=py -B5 -A5Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check the configuration model to understand the structure better
cat -n src/models/config.py | sed -n '1,100p'Repository: lightspeed-core/lightspeed-stack
Length of output: 3670
🏁 Script executed:
# Look for any schema or pydantic model definitions related to vector_stores
rg -n 'class.*VectorStore|vector_stores.*:' src/models/config.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Look for how YAML configuration is loaded and used
rg -n 'vector_stores.*default_provider_id|default_embedding_model' src --type=py -B5 -A5Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check if the top-level vector_stores section is part of the configuration model
rg -n 'vector_stores' src/models/config.py -C3Repository: lightspeed-core/lightspeed-stack
Length of output: 366
🏁 Script executed:
# Look for any YAML schema or documentation that describes the vector_stores structure
fd -e md -e rst | xargs rg -l 'vector_stores.*default_provider_id|default_provider_id.*vector' 2>/dev/nullRepository: lightspeed-core/lightspeed-stack
Length of output: 60
🏁 Script executed:
# Search in the entire codebase for any references to the top-level vector_stores config
rg -n 'registered_resources|vector_stores' tests/e2e/configs/run-bedrock.yaml -B2 -A2Repository: lightspeed-core/lightspeed-stack
Length of output: 585
🏁 Script executed:
# Look at the full bedrock config structure to understand better
cat -n tests/e2e/configs/run-bedrock.yaml | grep -E '^\s+[a-z_]+:|^[a-z_]+:' | head -40Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Search for where top-level (non-registered_resources) vector_stores is used
rg -n 'config.*vector_stores|vector_stores.*config' src --type=py -B2 -A2Repository: lightspeed-core/lightspeed-stack
Length of output: 2831
🏁 Script executed:
# Check if llama-stack loads and validates this top-level vector_stores section
rg -n 'yaml|load|parse' src/llama_stack_configuration.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 975
🏁 Script executed:
# Look at the entire structure around the top-level vector_stores in bedrock config
cat -n tests/e2e/configs/run-bedrock.yaml | sed -n '145,165p'Repository: lightspeed-core/lightspeed-stack
Length of output: 657
🏁 Script executed:
# Search for where top-level config keys are parsed (not in registered_resources)
rg -n 'safety|benchmarks' src --type=py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 1404
🏁 Script executed:
# Check if top-level vector_stores is actually used anywhere
rg -n '\[.vector_stores.\]|\[.default_provider_id.\]' src --type=py -B3 -A3Repository: lightspeed-core/lightspeed-stack
Length of output: 2799
🏁 Script executed:
# Look at the base config class to see what top-level keys are supported
cat src/models/config.py | grep -A200 'class AppConfig' | head -100Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check if the top-level vector_stores config is parsed/loaded by AppConfig
rg -n 'class AppConfig' src/models/config.py -A50 | head -80Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Look for any initialization or field definitions that include 'vector_stores' at top level
rg -n 'vector_stores.*=' src/models/config.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Search for whether top-level vector_stores is part of the configuration schema
python3 << 'EOF'
import sys
sys.path.insert(0, 'src')
try:
from models.config import AppConfig
import inspect
# Get all fields from AppConfig
if hasattr(AppConfig, 'model_fields'):
fields = AppConfig.model_fields
print("AppConfig fields:", list(fields.keys()))
if 'vector_stores' in fields:
print("vector_stores field found:", fields['vector_stores'])
else:
# Try to get annotations
if hasattr(AppConfig, '__annotations__'):
print("AppConfig annotations:", list(AppConfig.__annotations__.keys()))
except Exception as e:
print(f"Error: {e}")
import traceback
traceback.print_exc()
EOFRepository: lightspeed-core/lightspeed-stack
Length of output: 349
🏁 Script executed:
# Search for AppConfig class definition and its fields
rg -n 'class AppConfig' src/models/config.py -A100 | head -150Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Look for any mention of vector_stores or safety at top level
rg -n 'vector_stores:|safety:' src/models/config.pyRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check if the config allows extra/unknown fields (which would allow vector_stores at top level)
rg -n 'extra.*forbid|ConfigDict|model_config' src/models/config.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 144
🏁 Script executed:
# Let's check if top-level vector_stores is actually validated as part of AppConfig
rg -n 'AppConfig' src/models/config.py | head -5Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Find the AppConfig class definition in the file
cat src/models/config.py | tail -500 | head -200Repository: lightspeed-core/lightspeed-stack
Length of output: 6766
🏁 Script executed:
# Check if there's any post-load processing that handles the top-level vector_stores
rg -n 'def init|load.*config|parse.*config' src/models/config.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check if AppConfig strictly validates and rejects extra fields
rg -n -B5 -A30 'class AppConfig' src/models/config.py | tail -50Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Look for how the config is loaded and validated
rg -n 'AppConfig\|init_from' src/models/config.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check if there's a test that validates the bedrock config file
rg -n 'run-bedrock' tests --type=pyRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Let's see if the config has extra="forbid" and check all top-level fields in AppConfig
rg -n 'extra=' src/models/config.py -B2 -A2Repository: lightspeed-core/lightspeed-stack
Length of output: 387
🏁 Script executed:
# Find all top-level field definitions in AppConfig
rg -n ': ' src/models/config.py | grep -E '^\s+[a-z_]+: ' | tail -50Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check watsonx config to see if it has the same structure
cat tests/e2e/configs/run-watsonx.yaml | grep -A5 -B5 'vector_stores'Repository: lightspeed-core/lightspeed-stack
Length of output: 581
🏁 Script executed:
# Look at the entire AppConfig class definition
rg -n 'class AppConfig' src/models/config.py -A200 | head -250Repository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# Check if vector_stores is defined as a field in AppConfig
rg -n 'vector_stores' src/models/config.pyRepository: lightspeed-core/lightspeed-stack
Length of output: 182
🏁 Script executed:
# Check what the actual end of the file looks like
tail -100 src/models/config.pyRepository: lightspeed-core/lightspeed-stack
Length of output: 4599
🏁 Script executed:
# Look for where configs are loaded and if they're preprocessed before validation
rg -n 'AppConfig' src --type=py | grep -v test | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 1014
🏁 Script executed:
# Check if there's any code that removes or ignores top-level vector_stores
rg -n 'pop.*vector_stores|del.*vector_stores|remove.*vector_stores' src --type=pyRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
# See if AppConfig.init_from validates the config dict before loading
rg -n 'init_from|model_validate' src/models/config.py -B3 -A10 | head -50Repository: lightspeed-core/lightspeed-stack
Length of output: 58
Remove stale vector_stores section with non-existent FAISS provider.
The top-level vector_stores block (Lines 154-158) references default_provider_id: faiss, but providers.vector_io is now empty (Line 66). This orphaned configuration should be removed to avoid confusion or potential runtime issues if the top-level section is ever consumed by tooling or future code changes.
The same issue exists in tests/e2e/configs/run-watsonx.yaml (Lines 152-156).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/e2e/configs/run-bedrock.yaml` around lines 154 - 158, Remove the stale
top-level vector_stores block that references default_provider_id: faiss and the
default_embedding_model (provider_id: sentence-transformers, model_id:
all-mpnet-base-v2) from tests/e2e/configs/run-bedrock.yaml; also make the same
removal in tests/e2e/configs/run-watsonx.yaml to avoid an orphaned configuration
now that providers.vector_io is empty. Locate the block by the key vector_stores
and delete the entire mapping (default_provider_id and default_embedding_model)
so no FAISS reference remains.
| backend: kv_rag | ||
| provider_id: faiss | ||
| provider_type: inline::faiss | ||
| vector_io: [] |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
llama-stack run.yaml vector_stores default_provider_id missing provider validation
💡 Result:
In Llama Stack, the run.yaml configuration file defines providers and global settings for APIs including vector_io (vector stores). The vector_stores section sets defaults like default_provider_id, which specifies the ID of the vector store provider (e.g., "faiss") to use when not explicitly provided in API calls to /v1/vector_stores. If default_provider_id is missing or unset in run.yaml when using vector store APIs, the system lacks a default and may fail validation or routing, especially if multiple vector_io providers are configured. This leads to errors when creating vector stores without specifying "provider_id" in the extra_body parameter. To fix: 1. Ensure a vector_io provider is defined, e.g.: providers: vector_io: - provider_id: faiss provider_type: inline::faiss config: persistence: namespace: vector_io::faiss backend: kv_default 2. Add the vector_stores section with default_provider_id matching a defined provider_id: vector_stores: default_provider_id: faiss default_embedding_model: provider_id: ollama # or your embedding provider model_id: nomic-embed-text-v2-moe # example model Official docs confirm this structure (llamastack.github.io/docs/concepts/vector_stores_configuration). Examples in GitHub issues (#4047) show complete configs. No strict schema validation rejects missing fields at startup, but runtime vector store operations fail without defaults if provider_id unspecified (similar to provider routing errors in other APIs). Always specify provider_id explicitly in API calls as a best practice for multi-provider setups.
Citations:
- 1: bug: llama stack not erroring out when creating vector store with unknown embedding model ogx-ai/ogx#4047
- 2: https://llamastack.github.io/docs/concepts/vector_stores_configuration
- 3: https://llamastack.github.io/docs/distributions/configuration
- 4: https://github.com/lightspeed-core/lightspeed-stack/blob/main/run.yaml
- 5: https://github.com/tisnik/most-popular-python-libs/blob/master/llama-stack-demo/demo5/run.yaml
- 6: https://github.com/meta-llama/llama-stack
🏁 Script executed:
# First, let's locate the e2e config files
find . -path "*/e2e/configs/*.yaml" -type fRepository: lightspeed-core/lightspeed-stack
Length of output: 327
🏁 Script executed:
# Check the size and read run-vertexai.yaml
wc -l ./tests/e2e/configs/run-vertexai.yaml
echo "---"
cat -n ./tests/e2e/configs/run-vertexai.yamlRepository: lightspeed-core/lightspeed-stack
Length of output: 5183
🏁 Script executed:
# Check the other e2e configs to see which ones have the same issue
for config in ./tests/e2e/configs/run-*.yaml; do
echo "=== $(basename $config) ==="
echo "Line 67 (vector_io):"
sed -n '67p' "$config" || echo "Line 67 not found"
echo "vector_stores default_provider_id:"
grep -n "default_provider_id:" "$config" || echo "Not found"
echo "builtin::rag:"
grep -n "builtin::rag" "$config" || echo "Not found"
echo ""
doneRepository: lightspeed-core/lightspeed-stack
Length of output: 1640
🏁 Script executed:
# More thorough check: find vector_io sections in all configs
for config in ./tests/e2e/configs/run-*.yaml; do
echo "=== $(basename $config) ==="
# Extract the vector_io section (from "vector_io:" until next provider at same indent level)
grep -A 5 "^ vector_io:" "$config" || echo "vector_io: not found at section level"
echo ""
doneRepository: lightspeed-core/lightspeed-stack
Length of output: 1034
vector_io: [] leaves dangling faiss/RAG references downstream.
With no vector_io providers registered, the still-present vector_stores.default_provider_id: faiss (line 152) and the builtin::rag tool group registration (lines 148-150) reference a provider that no longer exists. This causes runtime failures the first time the RAG tool is invoked.
Either restore an inline vector_io provider (e.g., inline::faiss) and the matching vector_stores entry, or fully remove RAG: drop vector_io from apis, drop the builtin::rag tool group, and clear vector_stores.default_provider_id/default_embedding_model.
This inconsistency applies to all other e2e configs in this PR: run-azure.yaml, run-bedrock.yaml, run-ci.yaml, run-rhaiis.yaml, run-rhelai.yaml, and run-watsonx.yaml.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/e2e/configs/run-vertexai.yaml` at line 67, The config sets vector_io:
[] but still references a faiss provider via vector_stores.default_provider_id
and registers the builtin::rag tool group, causing runtime failures; either
re-add an inline vector_io provider (e.g., inline::faiss) and the corresponding
vector_stores entry including default_embedding_model, or remove RAG entirely by
deleting vector_io from apis, removing the builtin::rag tool group registration,
and clearing/omitting vector_stores.default_provider_id and
default_embedding_model; apply the same fix consistently across the other e2e
configs (run-azure.yaml, run-bedrock.yaml, run-ci.yaml, run-rhaiis.yaml,
run-rhelai.yaml, run-watsonx.yaml).
| # GET/DELETE: do not pass json= (even None); some stacks mishandle it. | ||
| # All methods: retry transient RST/connection errors after restarts in CI. | ||
| if method in ("GET", "DELETE"): | ||
| context.response = request_with_transient_retry( | ||
| method=method, url=url, headers=headers, timeout=DEFAULT_TIMEOUT | ||
| ) | ||
| else: | ||
| assert context.text is not None, "Payload needs to be specified" | ||
| data = json.loads(context.text) | ||
| context.response = request_with_transient_retry( | ||
| method=method, url=url, json=data, headers=headers, timeout=DEFAULT_TIMEOUT | ||
| ) |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Method-only branching may not cover all bodyless verbs.
The branching assumes only GET/DELETE are bodyless. A HEAD, OPTIONS, or even an intentionally-bodyless PUT/PATCH step would fall into the else branch and trip on assert context.text is not None. Since the existing .feature files in this repo presumably never use such methods, this is a low-risk concern, but consider keying off the presence of context.text rather than the verb to make the helper future-proof:
🔧 Suggested refactor
- if method in ("GET", "DELETE"):
- context.response = request_with_transient_retry(
- method=method, url=url, headers=headers, timeout=DEFAULT_TIMEOUT
- )
- else:
- assert context.text is not None, "Payload needs to be specified"
- data = json.loads(context.text)
- context.response = request_with_transient_retry(
- method=method, url=url, json=data, headers=headers, timeout=DEFAULT_TIMEOUT
- )
+ method_upper = method.upper()
+ kwargs: dict = {
+ "method": method_upper,
+ "url": url,
+ "headers": headers,
+ "timeout": DEFAULT_TIMEOUT,
+ }
+ if context.text is not None:
+ kwargs["json"] = json.loads(context.text)
+ elif method_upper not in ("GET", "DELETE", "HEAD", "OPTIONS"):
+ assert False, f"Payload needs to be specified for {method_upper}"
+ context.response = request_with_transient_retry(**kwargs)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/e2e/features/steps/common_http.py` around lines 225 - 236, The current
branching in the HTTP helper uses the HTTP verb check (method in ("GET",
"DELETE")) which misses other bodyless verbs and causes the assert on
context.text in functions like request_with_transient_retry to fail; change the
logic to decide whether to pass a JSON body based on presence of context.text
(e.g., if context.text is None then call request_with_transient_retry without
json=, otherwise json=json.loads(context.text)), keeping the same headers,
timeout (DEFAULT_TIMEOUT) and existing transient-retry behavior so HEAD/OPTIONS
or intentionally bodyless PUT/PATCH requests are handled correctly.
| def request_with_transient_retry( | ||
| **kwargs: Any, | ||
| ) -> requests.Response: | ||
| """Call :func:`requests.request`, retrying on transient :exc:`~requests.exceptions.ConnectionError`. | ||
|
|
||
| Parameters: | ||
| **kwargs: Forwarded to :func:`requests.request` (``method``, ``url``, ``json``, | ||
| ``headers``, ``timeout``, ``stream``, etc.). | ||
|
|
||
| Returns: | ||
| Successful :class:`requests.Response`. | ||
|
|
||
| Raises: | ||
| The last :exc:`requests.exceptions.ConnectionError` if all attempts fail. | ||
| """ | ||
| last_err: BaseException | None = None | ||
| for attempt in range(E2E_HTTP_TRANSIENT_MAX_ATTEMPTS): | ||
| try: | ||
| return requests.request(**kwargs) | ||
| except requests.exceptions.ConnectionError as exc: | ||
| last_err = exc | ||
| if attempt < E2E_HTTP_TRANSIENT_MAX_ATTEMPTS - 1: | ||
| time.sleep(E2E_HTTP_TRANSIENT_DELAY_S) | ||
| assert last_err is not None | ||
| raise last_err |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Retry semantics: ConnectionError retry replays non-idempotent requests.
request_with_transient_retry retries on any requests.exceptions.ConnectionError regardless of HTTP method. ConnectionError covers connection-reset/aborted-during-write scenarios where the server may have already received and partially processed the request. For non-idempotent POSTs (e.g., /v1/query, /v1/responses, /v1/conversations), this can result in duplicate work on the backend and confusing test state. Since this is e2e-only and aimed at "transient resets after container restarts", the trade-off is probably acceptable, but it's worth either:
- gating retries by method (only
GET/HEAD/DELETEby default), or - documenting the duplicate-side-effect risk in the docstring.
Additionally, last_err: BaseException | None is wider than necessary since only ConnectionError is captured — annotating as requests.exceptions.ConnectionError | None would express intent more precisely.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/e2e/utils/utils.py` around lines 68 - 92, The function
request_with_transient_retry currently retries on any
requests.exceptions.ConnectionError for all HTTP methods which can replay
non-idempotent requests (e.g., POST) and cause duplicate backend work; update
request_with_transient_retry to only retry for safe/idempotent methods (at
minimum allow retries for "GET", "HEAD", "DELETE") by checking
kwargs.get("method", "GET").upper() before the retry loop (use
E2E_HTTP_TRANSIENT_MAX_ATTEMPTS and E2E_HTTP_TRANSIENT_DELAY_S as before), or if
you prefer keep current behavior add a clear docstring note warning about
duplicate side-effects for non-idempotent methods; also narrow the last_err
annotation to requests.exceptions.ConnectionError | None to reflect the actual
captured exception type.
| for attempt in range(E2E_HTTP_TRANSIENT_MAX_ATTEMPTS): | ||
| try: | ||
| return requests.request(**kwargs) | ||
| except requests.exceptions.ConnectionError as exc: | ||
| last_err = exc | ||
| if attempt < E2E_HTTP_TRANSIENT_MAX_ATTEMPTS - 1: | ||
| time.sleep(E2E_HTTP_TRANSIENT_DELAY_S) |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Consider linear backoff and a final retry log.
Three attempts with a constant 0.5s delay (~1s total wait) is short for "container restarts" recovery (Llama Stack library mode boot is documented as ~45-60s elsewhere in this file, see restart_container/wait_for_container_health). Two minor refinements:
- Use exponential or at least linear backoff (
E2E_HTTP_TRANSIENT_DELAY_S * (attempt + 1)). - Log each retry (which attempt, last exception) so test logs surface transient-retry behavior rather than failing silently.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/e2e/utils/utils.py` around lines 84 - 90, Update the transient HTTP
retry loop that calls requests.request(**kwargs) (the loop using
E2E_HTTP_TRANSIENT_MAX_ATTEMPTS and E2E_HTTP_TRANSIENT_DELAY_S) to use
linear/exponential backoff (e.g., sleep E2E_HTTP_TRANSIENT_DELAY_S * (attempt +
1) or multiply by 2**attempt) and log each retry with the attempt index and
exception (use the module test logger or requests logger) so retries are
visible; also add a final error log before re-raising or returning the last
exception if all attempts fail. Ensure the changes reference the same retry loop
variables (E2E_HTTP_TRANSIENT_MAX_ATTEMPTS, E2E_HTTP_TRANSIENT_DELAY_S,
last_err) and preserve raising/return behavior.
tisnik
left a comment
There was a problem hiding this comment.
Seems correct. So for now the tests will run for each PR? What about token consumption?
|
|
||
| # Transient connection resets (e.g. errno 104) after container restarts in CI/Docker. | ||
| E2E_HTTP_TRANSIENT_MAX_ATTEMPTS: int = 3 | ||
| E2E_HTTP_TRANSIENT_DELAY_S: float = 0.5 |
There was a problem hiding this comment.
is it enough? 3*0.5 is pretty short time...
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/e2e_tests_providers.yaml:
- Around line 318-320: The step named "Remove the prefix for watsonx default
model" currently hardcodes meta-llama/llama-3-3-70b-instruct for
E2E_DEFAULT_MODEL_OVERRIDE; instead derive the value from the matrix entry that
contains the watsonx model id by stripping the leading "watsonx/" (or everything
up to the first slash) from that matrix variable and write that result to
E2E_DEFAULT_MODEL_OVERRIDE so the matrix remains the single source of truth
(update the run line in that step to compute and export the stripped matrix
value rather than the hardcoded string).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 4b0052bf-6dcb-4666-90c8-75a8ca8c4531
📒 Files selected for processing (3)
.github/workflows/e2e_tests_providers.yaml.github/workflows/e2e_tests_rhaiis.yaml.github/workflows/e2e_tests_rhelai.yaml
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)
- GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-on-pull-request
- GitHub Check: build-pr
- GitHub Check: unit_tests (3.12)
- GitHub Check: unit_tests (3.13)
- GitHub Check: integration_tests (3.13)
- GitHub Check: E2E: server mode / ci / group 2
- GitHub Check: E2E: library mode / ci / group 1
- GitHub Check: E2E: server mode / ci / group 1
- GitHub Check: Pylinter
- GitHub Check: E2E: library mode / ci / group 3
- GitHub Check: E2E: library mode / ci / group 2
- GitHub Check: E2E: server mode / ci / group 3
- GitHub Check: E2E Tests for Lightspeed Evaluation job
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-07-28T13:15:54.895Z
Learnt from: radofuchs
Repo: lightspeed-core/lightspeed-stack PR: 306
File: .github/workflows/e2e_tests.yaml:4-4
Timestamp: 2025-07-28T13:15:54.895Z
Learning: The user radofuchs prefers unrestricted pull_request_target workflows with secrets access for comprehensive PR testing, accepting the security trade-offs to ensure all problems are caught during the PR process.
Applied to files:
.github/workflows/e2e_tests_rhelai.yaml
| - name: Remove the prefix for watsonx default model | ||
| if: matrix.environment == 'watsonx' | ||
| run: echo "E2E_DEFAULT_MODEL_OVERRIDE=meta-llama/llama-3-3-70b-instruct" >> $GITHUB_ENV |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Avoid hardcoding the unprefixed watsonx model — derive it from the matrix value.
Line 320 hardcodes meta-llama/llama-3-3-70b-instruct, duplicating the watsonx model id that already lives on Line 32 (watsonx/meta-llama/llama-3-3-70b-instruct). If the matrix entry is ever updated (different llama version, different watsonx model), this step will silently set E2E_DEFAULT_MODEL_OVERRIDE to a stale/wrong value and the e2e suite will run against a model that isn't the one registered/connectivity-checked.
Strip the prefix from the env var the matrix already populates instead:
♻️ Proposed refactor
- name: Remove the prefix for watsonx default model
if: matrix.environment == 'watsonx'
- run: echo "E2E_DEFAULT_MODEL_OVERRIDE=meta-llama/llama-3-3-70b-instruct" >> $GITHUB_ENV
+ run: echo "E2E_DEFAULT_MODEL_OVERRIDE=${E2E_DEFAULT_MODEL_OVERRIDE#watsonx/}" >> "$GITHUB_ENV"This keeps the matrix on Line 32 as the single source of truth for the watsonx model identifier.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e_tests_providers.yaml around lines 318 - 320, The step
named "Remove the prefix for watsonx default model" currently hardcodes
meta-llama/llama-3-3-70b-instruct for E2E_DEFAULT_MODEL_OVERRIDE; instead derive
the value from the matrix entry that contains the watsonx model id by stripping
the leading "watsonx/" (or everything up to the first slash) from that matrix
variable and write that result to E2E_DEFAULT_MODEL_OVERRIDE so the matrix
remains the single source of truth (update the run line in that step to compute
and export the stripped matrix value rather than the hardcoded string).
…ore#1596) * fix providers tests * revert triggers to run on schedule --------- Co-authored-by: Radovan Fuchs <[email protected]>
Description
Get the providers e2e tests to run properly.
See https://github.com/are-ces/lightspeed-stack/actions/runs/24983078532/job/73149647408
Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit
Tests
Chores