Thanks to visit codestin.com
Credit goes to github.com

Skip to content

LCORE-1249: stub for benchmarks#1084

Merged
tisnik merged 2 commits into
lightspeed-core:mainfrom
tisnik:lcore-1249-benchmarks-stub
Feb 1, 2026
Merged

LCORE-1249: stub for benchmarks#1084
tisnik merged 2 commits into
lightspeed-core:mainfrom
tisnik:lcore-1249-benchmarks-stub

Conversation

@tisnik

@tisnik tisnik commented Feb 1, 2026

Copy link
Copy Markdown
Contributor

Description

LCORE-1249: stub for benchmarks

Type of change

  • Refactor
  • New feature
  • Bug fix
  • CVE fix
  • Optimization
  • Documentation Update
  • Configuration Update
  • Bump-up service version
  • Bump-up dependent library
  • Bump-up library or tool used for development (does not change the final image)
  • CI configuration change
  • Konflux configuration change
  • Unit tests improvement
  • Integration tests improvement
  • End to end tests improvement
  • Benchmarks

Tools used to create PR

  • Assisted-by: N/A
  • Generated by: N/A

Related Tickets & Documents

  • Related Issue #LCORE-1249

Summary by CodeRabbit

  • Tests

    • Added comprehensive benchmark tests measuring storing, updating, and listing user conversations across small, medium, and large dataset scenarios.
  • Documentation

    • Added benchmark documentation and a README entry describing "Benchmarks for the LCORE".
  • Chores

    • Added a benchmark environment configuration to support local SQLite-backed performance runs.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai

coderabbitai Bot commented Feb 1, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Adds a pytest-benchmark suite and supporting config for LCORE: a new benchmark test module with fixtures, data generators, and benchmarks measuring store, update, and list operations on an SQLite-backed UserConversation table; plus a YAML config and README line.

Changes

Cohort / File(s) Summary
Benchmark tests
tests/benchmarks/test_app_database.py
New pytest-benchmark module: fixtures to load YAML configuration and create an in-memory/temporary SQLite DB, utilities to generate provider/model/topic summaries, CRUD helpers to store/update/list UserConversation rows, and 9 benchmark entry tests covering small (0), middle (1,000), and large (10,000) datasets. Review DB session handling, commits, and pre-population logic.
Benchmark README
tests/benchmarks/README.md
Adds single line "Benchmarks for the LCORE" to README; documentation-only change.
Benchmark configuration
tests/configuration/benchmarks-lightspeed-stack.yaml
Adds YAML configuration for Lightspeed Core Service (service, llama_stack integration, user data collection, sqlite database and conversation_cache paths, noop auth). No runtime code changes, but tests depend on expected keys/paths in this file.

Sequence Diagram(s)

sequenceDiagram
    participant Tester as Test Runner
    participant Config as Config Loader
    participant DBInit as DB Initializer
    participant DB as SQLite DB
    participant Benchmark as Benchmark Loop

    Tester->>Config: load YAML (benchmarks-lightspeed-stack.yaml)
    Tester->>DBInit: request temporary DB path & tables
    DBInit->>DB: create tables / session factory
    Tester->>DB: pre-populate N rows (0/1000/10000)
    Tester->>Benchmark: invoke benchmark function
    Benchmark->>DB: perform operation (store/update/list) repeatedly
    Benchmark->>Tester: return timing results
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'LCORE-1249: stub for benchmarks' clearly summarizes the main change—adding a benchmark stub implementation for LCORE-1249, which aligns with the changeset that introduces benchmark test files, configuration, and documentation.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@tisnik tisnik force-pushed the lcore-1249-benchmarks-stub branch from 6cc3fe1 to 0b55ed9 Compare February 1, 2026 10:39

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@tests/benchmarks/test_app_database.py`:
- Around line 258-262: The docstring for the function that accepts parameters
session (Session) and id currently describes id as optional, but the signature
requires id: str; update the docstring to reflect that id is required (id (str):
explicit ID to assign to the new conversation) and remove any wording about it
being optional or a generated suid; ensure the parameter types and description
match the signature exactly (reference the parameter name "id" and the "session
(Session)" parameter to locate the docstring).
- Around line 1-10: Reorder the import block in the test module so standard
library imports (datetime, random, pathlib.Path, typing.Optional) come first,
followed by third-party imports (pytest, sqlalchemy.orm.Session,
pytest_benchmark.fixture.BenchmarkFixture), then any local imports; specifically
adjust the imports shown (datetime/UTC, datetime, random, Path, Optional,
pytest, Session, BenchmarkFixture) into that grouping and run the formatter (uv
run make format) to apply Black/isort rules and commit the result.
🧹 Nitpick comments (1)
tests/benchmarks/README.md (1)

1-1: Make this a Markdown heading for better rendering.

📌 Suggested tweak
-Benchmarks for the LCORE
+# Benchmarks for the LCORE

Comment on lines +1 to +10
"""Benchmarks for app.database module."""

from typing import Optional
from pathlib import Path
import pytest
import random
from sqlalchemy.orm import Session
from pytest_benchmark.fixture import BenchmarkFixture
from datetime import UTC, datetime

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if the file exists and its current state
find . -name "test_app_database.py" -type f

Repository: lightspeed-core/lightspeed-stack

Length of output: 115


🏁 Script executed:

# Check the first 15 lines of the file to see current state
if [ -f "tests/benchmarks/test_app_database.py" ]; then
  echo "=== Current file content (first 15 lines) ==="
  head -15 "tests/benchmarks/test_app_database.py"
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 563


🏁 Script executed:

# Check if uv and make are available, and what format command does
if command -v uv &> /dev/null; then
  echo "=== Checking for make format command ==="
  cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
  if [ -f "Makefile" ]; then
    grep -A 5 "^format:" Makefile || echo "No format target found"
  fi
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 58


🏁 Script executed:

#!/bin/bash
# Find Makefile and check format target
find . -name "Makefile" -o -name "pyproject.toml" -o -name ".black" -o -name ".isort.cfg" | head -20

Repository: lightspeed-core/lightspeed-stack

Length of output: 103


🏁 Script executed:

# Check if pyproject.toml exists and look for tool configuration
if [ -f "pyproject.toml" ]; then
  echo "=== pyproject.toml format/black/isort sections ==="
  grep -A 20 "\[tool.black\]\|\[tool.isort\]\|\[tool.ruff\]" pyproject.toml | head -40
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 520


🏁 Script executed:

# Try to run Black formatter to see what it would change
if command -v black &> /dev/null; then
  echo "=== Black would format to: ==="
  black --diff tests/benchmarks/test_app_database.py 2>&1 | head -50
elif command -v uv &> /dev/null; then
  echo "=== Checking if uv can run black ==="
  uv run black --version 2>&1 || echo "uv black not available"
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 58


🏁 Script executed:

#!/bin/bash
# Read the Makefile to find the format target
cat Makefile | grep -A 5 "^format:"

Repository: lightspeed-core/lightspeed-stack

Length of output: 267


🏁 Script executed:

#!/bin/bash
# Try to run the format command as specified in learning: uv run make format
cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
if command -v uv &> /dev/null && [ -f "Makefile" ]; then
  echo "=== Running: uv run make format (on test file) ==="
  # First, check if make format would work
  make format 2>&1 | head -30
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 58


🏁 Script executed:

#!/bin/bash
# If make format is not available, try Black directly with diff
if command -v black &> /dev/null; then
  echo "=== Black formatting diff for test_app_database.py ==="
  black --diff --target-version py311 tests/benchmarks/test_app_database.py 2>&1
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 58


🏁 Script executed:

#!/bin/bash
# Try to check what ruff and isort think about the imports
# First, let's see if we can analyze the imports manually

echo "=== Current import order in test_app_database.py ==="
head -15 tests/benchmarks/test_app_database.py | grep -n "^from\|^import"

echo ""
echo "=== What the correct order should be (stdlib, third-party, local) ==="
cat << 'EOF'
Expected order:
1. Docstring
2. Standard library: datetime, pathlib, random, typing
3. Third-party: pytest, sqlalchemy, pytest_benchmark
4. Local: app, configuration, utils, models
EOF

Repository: lightspeed-core/lightspeed-stack

Length of output: 807


🏁 Script executed:

#!/bin/bash
# Check ruff configuration for import sorting
if [ -f "pyproject.toml" ]; then
  echo "=== Checking ruff isort config ==="
  grep -A 10 "\[tool.ruff.isort\]" pyproject.toml || echo "No explicit isort config in ruff"
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 143


🏁 Script executed:

#!/bin/bash
# Let's manually verify the import organization problem
python3 << 'EOFPYTHON'
import re

with open('tests/benchmarks/test_app_database.py', 'r') as f:
    lines = f.readlines()

# Extract imports section (before blank line after imports)
docstring_end = 0
for i, line in enumerate(lines):
    if '"""' in line and i > 0:
        docstring_end = i + 1
        break

import_lines = []
stdlib_imports = []
thirdparty_imports = []
local_imports = []

for i in range(docstring_end, len(lines)):
    line = lines[i].strip()
    if not line or line.startswith('#'):
        if import_lines:  # End of import block
            break
        continue
    if line.startswith('from ') or line.startswith('import '):
        import_lines.append((i, line))
        
        # Classify imports
        if any(x in line for x in ['datetime', 'typing', 'pathlib', 'random']):
            stdlib_imports.append((i, line))
        elif any(x in line for x in ['pytest', 'sqlalchemy', 'pytest_benchmark']):
            thirdparty_imports.append((i, line))
        elif any(x in line for x in ['app', 'configuration', 'utils', 'models']):
            local_imports.append((i, line))
        else:
            print(f"Unclassified: {line}")

print("=== Import Classification ===")
print("\nStandard Library Imports:")
for line_no, imp in stdlib_imports:
    print(f"  Line {line_no+1}: {imp}")

print("\nThird-party Imports:")
for line_no, imp in thirdparty_imports:
    print(f"  Line {line_no+1}: {imp}")

print("\nLocal Imports:")
for line_no, imp in local_imports:
    print(f"  Line {line_no+1}: {imp}")

print("\n=== ISSUE ===")
print("Imports are NOT properly grouped by category.")
print("Black uses isort for import sorting, which requires:")
print("1. Standard library first")
print("2. Third-party packages next")
print("3. Local imports last")
EOFPYTHON

Repository: lightspeed-core/lightspeed-stack

Length of output: 361


🏁 Script executed:

#!/bin/bash
# Let's directly check what imports would be sorted to by analyzing the actual order
cat tests/benchmarks/test_app_database.py | head -20

Repository: lightspeed-core/lightspeed-stack

Length of output: 644


🏁 Script executed:

#!/bin/bash
# Verify the exact issue: imports are out of order
# datetime and random (stdlib) come after third-party imports
echo "=== ISSUE FOUND ==="
echo "Current order (lines 3-9):"
echo "  3: from typing import Optional         [stdlib - OK]"
echo "  4: from pathlib import Path             [stdlib - OK]"
echo "  5: import pytest                        [third-party - WRONG! Should be after ALL stdlib]"
echo "  6: import random                        [stdlib - WRONG! Should come before line 5]"
echo "  7: from sqlalchemy.orm import Session   [third-party]"
echo "  8: from pytest_benchmark.fixture...     [third-party]"
echo "  9: from datetime import UTC, datetime   [stdlib - WRONG! Should come before line 5]"
echo ""
echo "Correct order should be:"
echo "  1. Docstring"
echo "  2. from datetime import UTC, datetime"
echo "  3. from pathlib import Path"
echo "  4. import random"
echo "  5. from typing import Optional"
echo "  6. [blank line]"
echo "  7. import pytest"
echo "  8. from pytest_benchmark.fixture import BenchmarkFixture"
echo "  9. from sqlalchemy.orm import Session"
echo " 10. [blank line]"
echo " 11-15. Local imports"

Repository: lightspeed-core/lightspeed-stack

Length of output: 948


Fix import ordering to pass Black formatting.

The imports are not properly grouped by category. Black/isort requires imports to be ordered as: standard library, third-party packages, then local imports. Currently, datetime and random (stdlib) are mixed with third-party packages (pytest, sqlalchemy, pytest_benchmark).

Run uv run make format to automatically sort imports and reformat the file, then commit the changes.

🧰 Tools
🪛 GitHub Actions: Black

[error] 1-1: Black formatting check failed. 1 file would be reformatted. Run 'black' to fix formatting.

🤖 Prompt for AI Agents
In `@tests/benchmarks/test_app_database.py` around lines 1 - 10, Reorder the
import block in the test module so standard library imports (datetime, random,
pathlib.Path, typing.Optional) come first, followed by third-party imports
(pytest, sqlalchemy.orm.Session, pytest_benchmark.fixture.BenchmarkFixture),
then any local imports; specifically adjust the imports shown (datetime/UTC,
datetime, random, Path, Optional, pytest, Session, BenchmarkFixture) into that
grouping and run the formatter (uv run make format) to apply Black/isort rules
and commit the result.

Comment on lines +258 to +262
Parameters:
session (Session): SQLAlchemy session used to persist the record.
id (Optional[str]): Optional explicit ID to assign to the new conversation.
If not provided, a generated suid will be used.

@coderabbitai coderabbitai Bot Feb 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Docstring doesn’t match the function signature.
id is required (str), but the docstring describes it as optional.

✏️ Suggested docstring fix
-        id (Optional[str]): Optional explicit ID to assign to the new conversation.
-            If not provided, a generated suid will be used.
+        id (str): ID of the conversation to update.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Parameters:
session (Session): SQLAlchemy session used to persist the record.
id (Optional[str]): Optional explicit ID to assign to the new conversation.
If not provided, a generated suid will be used.
Parameters:
session (Session): SQLAlchemy session used to persist the record.
id (str): ID of the conversation to update.
🤖 Prompt for AI Agents
In `@tests/benchmarks/test_app_database.py` around lines 258 - 262, The docstring
for the function that accepts parameters session (Session) and id currently
describes id as optional, but the signature requires id: str; update the
docstring to reflect that id is required (id (str): explicit ID to assign to the
new conversation) and remove any wording about it being optional or a generated
suid; ensure the parameter types and description match the signature exactly
(reference the parameter name "id" and the "session (Session)" parameter to
locate the docstring).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai create issue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue for the docstring mismatch

gh issue create \
  --title "Fix docstring for update_user_conversation function in test_app_database.py" \
  --body "The docstring for the \`update_user_conversation\` function in \`tests/benchmarks/test_app_database.py\` incorrectly describes the \`id\` parameter as optional, but the function signature shows it as required (\`id: str\`).

The docstring should be updated to reflect that \`id\` is a required parameter.

**Related PR:** `#1084`
**Comment:** https://github.com/lightspeed-core/lightspeed-stack/pull/1084#discussion_r2751030268
**Requested by:** `@tisnik`" \
  --assignee tisnik

Length of output: 63


@tisnik, I've created issue #1086 to track the docstring fix for the update_user_conversation function.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@tests/benchmarks/test_app_database.py`:
- Around line 52-61: The test mutates global
configuration.database_configuration.sqlite.db_path which is fragile for
parallel tests; instead, avoid changing global state by using a test-scoped
configuration or injection: create a local test config (or use monkeypatch) to
set sqlite.db_path from tmp_path and pass that into
database.initialize_database/create_tables (or refactor initialize_database to
accept a db_path parameter), ensuring you reference and update the test-local
copy rather than configuration.database_configuration.sqlite.db_path so each
test gets its own isolated DB without touching globals.
🧹 Nitpick comments (3)
tests/benchmarks/test_app_database.py (3)

64-82: Consider seeding random for reproducible benchmarks.

The benchmark uses random.choice() throughout but never seeds the RNG. This means benchmark runs may produce slightly different data distributions, which could affect reproducibility when comparing results across runs.

Consider adding random.seed(SOME_CONSTANT) in the sqlite_database_fixture or at module level.

♻️ Proposed fix to add random seeding in fixture
 def sqlite_database_fixture(configuration_filename: str, tmp_path: Path) -> None:
     ...
     # try to load the configuration containing SQLite database setup
     configuration.load_configuration(configuration_filename)
     assert configuration.database_configuration.sqlite is not None
 
+    # Seed random for reproducible benchmark data
+    random.seed(42)
+
     # we need to start each benchmark with empty database
     configuration.database_configuration.sqlite.db_path = str(tmp_path / "database.db")

294-298: Redundant assertion: len() >= 0 is always true.

The assertion on line 298 is a tautology—len() always returns a non-negative integer. Consider removing it or replacing with a more meaningful check if needed.

🧹 Proposed fix
     user_conversations = query.all()
     assert user_conversations is not None
-    assert len(user_conversations) >= 0

316-321: Variable id shadows the builtin.

Using id as a loop variable (also at lines 393, 465) shadows Python's built-in id() function. While harmless here, consider using record_id or idx for clarity.

🧹 Proposed fix
-        for id in range(records_to_insert):
-            store_new_user_conversation(session, str(id))
+        for record_id in range(records_to_insert):
+            store_new_user_conversation(session, str(record_id))

Comment on lines +52 to +61
# try to load the configuration containing SQLite database setup
configuration.load_configuration(configuration_filename)
assert configuration.database_configuration.sqlite is not None

# we need to start each benchmark with empty database
configuration.database_configuration.sqlite.db_path = str(tmp_path / "database.db")

# initialize database session and create tables
database.initialize_database()
database.create_tables()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if pytest-xdist or parallel test configuration exists
echo "=== Checking for parallel test configuration in config files ==="
find . -type f \( -name "pytest.ini" -o -name "setup.cfg" -o -name "pyproject.toml" -o -name "tox.ini" -o -name "conftest.py" \) | head -20 | while read f; do
  echo "--- File: $f ---"
  if grep -q "xdist\|addopts.*-n\|parallel\|looponfail" "$f" 2>/dev/null; then
    grep -n "xdist\|addopts.*-n\|parallel\|looponfail" "$f" || true
  fi
done

echo ""
echo "=== Checking for pytest markers in benchmarks directory ==="
rg "pytest.mark" tests/benchmarks/ -A 2 || echo "No special markers found"

echo ""
echo "=== Examining the test file structure ==="
head -70 tests/benchmarks/test_app_database.py | tail -50

Repository: lightspeed-core/lightspeed-stack

Length of output: 2161


The global configuration mutation pattern limits flexibility for future parallel execution.

The fixture mutates configuration.database_configuration.sqlite.db_path directly on a global object (line 57). While benchmarks don't currently run in parallel, this pattern prevents future parallelization without refactoring. The use of tmp_path correctly isolates database files per test, but modifying global configuration state is fragile.

Either add documentation explicitly stating benchmarks must run sequentially, or refactor the fixture to avoid mutating global state (e.g., using test-scoped configuration instances or dependency injection).

🤖 Prompt for AI Agents
In `@tests/benchmarks/test_app_database.py` around lines 52 - 61, The test mutates
global configuration.database_configuration.sqlite.db_path which is fragile for
parallel tests; instead, avoid changing global state by using a test-scoped
configuration or injection: create a local test config (or use monkeypatch) to
set sqlite.db_path from tmp_path and pass that into
database.initialize_database/create_tables (or refactor initialize_database to
accept a db_path parameter), ensuring you reference and update the test-local
copy rather than configuration.database_configuration.sqlite.db_path so each
test gets its own isolated DB without touching globals.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant