LCORE-1249: stub for benchmarks#1084
Conversation
WalkthroughAdds a pytest-benchmark suite and supporting config for LCORE: a new benchmark test module with fixtures, data generators, and benchmarks measuring store, update, and list operations on an SQLite-backed UserConversation table; plus a YAML config and README line. Changes
Sequence Diagram(s)sequenceDiagram
participant Tester as Test Runner
participant Config as Config Loader
participant DBInit as DB Initializer
participant DB as SQLite DB
participant Benchmark as Benchmark Loop
Tester->>Config: load YAML (benchmarks-lightspeed-stack.yaml)
Tester->>DBInit: request temporary DB path & tables
DBInit->>DB: create tables / session factory
Tester->>DB: pre-populate N rows (0/1000/10000)
Tester->>Benchmark: invoke benchmark function
Benchmark->>DB: perform operation (store/update/list) repeatedly
Benchmark->>Tester: return timing results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
6cc3fe1 to
0b55ed9
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@tests/benchmarks/test_app_database.py`:
- Around line 258-262: The docstring for the function that accepts parameters
session (Session) and id currently describes id as optional, but the signature
requires id: str; update the docstring to reflect that id is required (id (str):
explicit ID to assign to the new conversation) and remove any wording about it
being optional or a generated suid; ensure the parameter types and description
match the signature exactly (reference the parameter name "id" and the "session
(Session)" parameter to locate the docstring).
- Around line 1-10: Reorder the import block in the test module so standard
library imports (datetime, random, pathlib.Path, typing.Optional) come first,
followed by third-party imports (pytest, sqlalchemy.orm.Session,
pytest_benchmark.fixture.BenchmarkFixture), then any local imports; specifically
adjust the imports shown (datetime/UTC, datetime, random, Path, Optional,
pytest, Session, BenchmarkFixture) into that grouping and run the formatter (uv
run make format) to apply Black/isort rules and commit the result.
🧹 Nitpick comments (1)
tests/benchmarks/README.md (1)
1-1: Make this a Markdown heading for better rendering.📌 Suggested tweak
-Benchmarks for the LCORE +# Benchmarks for the LCORE
| """Benchmarks for app.database module.""" | ||
|
|
||
| from typing import Optional | ||
| from pathlib import Path | ||
| import pytest | ||
| import random | ||
| from sqlalchemy.orm import Session | ||
| from pytest_benchmark.fixture import BenchmarkFixture | ||
| from datetime import UTC, datetime | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if the file exists and its current state
find . -name "test_app_database.py" -type fRepository: lightspeed-core/lightspeed-stack
Length of output: 115
🏁 Script executed:
# Check the first 15 lines of the file to see current state
if [ -f "tests/benchmarks/test_app_database.py" ]; then
echo "=== Current file content (first 15 lines) ==="
head -15 "tests/benchmarks/test_app_database.py"
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 563
🏁 Script executed:
# Check if uv and make are available, and what format command does
if command -v uv &> /dev/null; then
echo "=== Checking for make format command ==="
cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
if [ -f "Makefile" ]; then
grep -A 5 "^format:" Makefile || echo "No format target found"
fi
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
#!/bin/bash
# Find Makefile and check format target
find . -name "Makefile" -o -name "pyproject.toml" -o -name ".black" -o -name ".isort.cfg" | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 103
🏁 Script executed:
# Check if pyproject.toml exists and look for tool configuration
if [ -f "pyproject.toml" ]; then
echo "=== pyproject.toml format/black/isort sections ==="
grep -A 20 "\[tool.black\]\|\[tool.isort\]\|\[tool.ruff\]" pyproject.toml | head -40
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 520
🏁 Script executed:
# Try to run Black formatter to see what it would change
if command -v black &> /dev/null; then
echo "=== Black would format to: ==="
black --diff tests/benchmarks/test_app_database.py 2>&1 | head -50
elif command -v uv &> /dev/null; then
echo "=== Checking if uv can run black ==="
uv run black --version 2>&1 || echo "uv black not available"
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
#!/bin/bash
# Read the Makefile to find the format target
cat Makefile | grep -A 5 "^format:"Repository: lightspeed-core/lightspeed-stack
Length of output: 267
🏁 Script executed:
#!/bin/bash
# Try to run the format command as specified in learning: uv run make format
cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
if command -v uv &> /dev/null && [ -f "Makefile" ]; then
echo "=== Running: uv run make format (on test file) ==="
# First, check if make format would work
make format 2>&1 | head -30
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
#!/bin/bash
# If make format is not available, try Black directly with diff
if command -v black &> /dev/null; then
echo "=== Black formatting diff for test_app_database.py ==="
black --diff --target-version py311 tests/benchmarks/test_app_database.py 2>&1
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 58
🏁 Script executed:
#!/bin/bash
# Try to check what ruff and isort think about the imports
# First, let's see if we can analyze the imports manually
echo "=== Current import order in test_app_database.py ==="
head -15 tests/benchmarks/test_app_database.py | grep -n "^from\|^import"
echo ""
echo "=== What the correct order should be (stdlib, third-party, local) ==="
cat << 'EOF'
Expected order:
1. Docstring
2. Standard library: datetime, pathlib, random, typing
3. Third-party: pytest, sqlalchemy, pytest_benchmark
4. Local: app, configuration, utils, models
EOFRepository: lightspeed-core/lightspeed-stack
Length of output: 807
🏁 Script executed:
#!/bin/bash
# Check ruff configuration for import sorting
if [ -f "pyproject.toml" ]; then
echo "=== Checking ruff isort config ==="
grep -A 10 "\[tool.ruff.isort\]" pyproject.toml || echo "No explicit isort config in ruff"
fiRepository: lightspeed-core/lightspeed-stack
Length of output: 143
🏁 Script executed:
#!/bin/bash
# Let's manually verify the import organization problem
python3 << 'EOFPYTHON'
import re
with open('tests/benchmarks/test_app_database.py', 'r') as f:
lines = f.readlines()
# Extract imports section (before blank line after imports)
docstring_end = 0
for i, line in enumerate(lines):
if '"""' in line and i > 0:
docstring_end = i + 1
break
import_lines = []
stdlib_imports = []
thirdparty_imports = []
local_imports = []
for i in range(docstring_end, len(lines)):
line = lines[i].strip()
if not line or line.startswith('#'):
if import_lines: # End of import block
break
continue
if line.startswith('from ') or line.startswith('import '):
import_lines.append((i, line))
# Classify imports
if any(x in line for x in ['datetime', 'typing', 'pathlib', 'random']):
stdlib_imports.append((i, line))
elif any(x in line for x in ['pytest', 'sqlalchemy', 'pytest_benchmark']):
thirdparty_imports.append((i, line))
elif any(x in line for x in ['app', 'configuration', 'utils', 'models']):
local_imports.append((i, line))
else:
print(f"Unclassified: {line}")
print("=== Import Classification ===")
print("\nStandard Library Imports:")
for line_no, imp in stdlib_imports:
print(f" Line {line_no+1}: {imp}")
print("\nThird-party Imports:")
for line_no, imp in thirdparty_imports:
print(f" Line {line_no+1}: {imp}")
print("\nLocal Imports:")
for line_no, imp in local_imports:
print(f" Line {line_no+1}: {imp}")
print("\n=== ISSUE ===")
print("Imports are NOT properly grouped by category.")
print("Black uses isort for import sorting, which requires:")
print("1. Standard library first")
print("2. Third-party packages next")
print("3. Local imports last")
EOFPYTHONRepository: lightspeed-core/lightspeed-stack
Length of output: 361
🏁 Script executed:
#!/bin/bash
# Let's directly check what imports would be sorted to by analyzing the actual order
cat tests/benchmarks/test_app_database.py | head -20Repository: lightspeed-core/lightspeed-stack
Length of output: 644
🏁 Script executed:
#!/bin/bash
# Verify the exact issue: imports are out of order
# datetime and random (stdlib) come after third-party imports
echo "=== ISSUE FOUND ==="
echo "Current order (lines 3-9):"
echo " 3: from typing import Optional [stdlib - OK]"
echo " 4: from pathlib import Path [stdlib - OK]"
echo " 5: import pytest [third-party - WRONG! Should be after ALL stdlib]"
echo " 6: import random [stdlib - WRONG! Should come before line 5]"
echo " 7: from sqlalchemy.orm import Session [third-party]"
echo " 8: from pytest_benchmark.fixture... [third-party]"
echo " 9: from datetime import UTC, datetime [stdlib - WRONG! Should come before line 5]"
echo ""
echo "Correct order should be:"
echo " 1. Docstring"
echo " 2. from datetime import UTC, datetime"
echo " 3. from pathlib import Path"
echo " 4. import random"
echo " 5. from typing import Optional"
echo " 6. [blank line]"
echo " 7. import pytest"
echo " 8. from pytest_benchmark.fixture import BenchmarkFixture"
echo " 9. from sqlalchemy.orm import Session"
echo " 10. [blank line]"
echo " 11-15. Local imports"Repository: lightspeed-core/lightspeed-stack
Length of output: 948
Fix import ordering to pass Black formatting.
The imports are not properly grouped by category. Black/isort requires imports to be ordered as: standard library, third-party packages, then local imports. Currently, datetime and random (stdlib) are mixed with third-party packages (pytest, sqlalchemy, pytest_benchmark).
Run uv run make format to automatically sort imports and reformat the file, then commit the changes.
🧰 Tools
🪛 GitHub Actions: Black
[error] 1-1: Black formatting check failed. 1 file would be reformatted. Run 'black' to fix formatting.
🤖 Prompt for AI Agents
In `@tests/benchmarks/test_app_database.py` around lines 1 - 10, Reorder the
import block in the test module so standard library imports (datetime, random,
pathlib.Path, typing.Optional) come first, followed by third-party imports
(pytest, sqlalchemy.orm.Session, pytest_benchmark.fixture.BenchmarkFixture),
then any local imports; specifically adjust the imports shown (datetime/UTC,
datetime, random, Path, Optional, pytest, Session, BenchmarkFixture) into that
grouping and run the formatter (uv run make format) to apply Black/isort rules
and commit the result.
| Parameters: | ||
| session (Session): SQLAlchemy session used to persist the record. | ||
| id (Optional[str]): Optional explicit ID to assign to the new conversation. | ||
| If not provided, a generated suid will be used. | ||
|
|
There was a problem hiding this comment.
Docstring doesn’t match the function signature.
id is required (str), but the docstring describes it as optional.
✏️ Suggested docstring fix
- id (Optional[str]): Optional explicit ID to assign to the new conversation.
- If not provided, a generated suid will be used.
+ id (str): ID of the conversation to update.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Parameters: | |
| session (Session): SQLAlchemy session used to persist the record. | |
| id (Optional[str]): Optional explicit ID to assign to the new conversation. | |
| If not provided, a generated suid will be used. | |
| Parameters: | |
| session (Session): SQLAlchemy session used to persist the record. | |
| id (str): ID of the conversation to update. | |
🤖 Prompt for AI Agents
In `@tests/benchmarks/test_app_database.py` around lines 258 - 262, The docstring
for the function that accepts parameters session (Session) and id currently
describes id as optional, but the signature requires id: str; update the
docstring to reflect that id is required (id (str): explicit ID to assign to the
new conversation) and remove any wording about it being optional or a generated
suid; ensure the parameter types and description match the signature exactly
(reference the parameter name "id" and the "session (Session)" parameter to
locate the docstring).
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Create a GitHub issue for the docstring mismatch
gh issue create \
--title "Fix docstring for update_user_conversation function in test_app_database.py" \
--body "The docstring for the \`update_user_conversation\` function in \`tests/benchmarks/test_app_database.py\` incorrectly describes the \`id\` parameter as optional, but the function signature shows it as required (\`id: str\`).
The docstring should be updated to reflect that \`id\` is a required parameter.
**Related PR:** `#1084`
**Comment:** https://github.com/lightspeed-core/lightspeed-stack/pull/1084#discussion_r2751030268
**Requested by:** `@tisnik`" \
--assignee tisnik
Length of output: 63
@tisnik, I've created issue #1086 to track the docstring fix for the update_user_conversation function.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@tests/benchmarks/test_app_database.py`:
- Around line 52-61: The test mutates global
configuration.database_configuration.sqlite.db_path which is fragile for
parallel tests; instead, avoid changing global state by using a test-scoped
configuration or injection: create a local test config (or use monkeypatch) to
set sqlite.db_path from tmp_path and pass that into
database.initialize_database/create_tables (or refactor initialize_database to
accept a db_path parameter), ensuring you reference and update the test-local
copy rather than configuration.database_configuration.sqlite.db_path so each
test gets its own isolated DB without touching globals.
🧹 Nitpick comments (3)
tests/benchmarks/test_app_database.py (3)
64-82: Consider seedingrandomfor reproducible benchmarks.The benchmark uses
random.choice()throughout but never seeds the RNG. This means benchmark runs may produce slightly different data distributions, which could affect reproducibility when comparing results across runs.Consider adding
random.seed(SOME_CONSTANT)in thesqlite_database_fixtureor at module level.♻️ Proposed fix to add random seeding in fixture
def sqlite_database_fixture(configuration_filename: str, tmp_path: Path) -> None: ... # try to load the configuration containing SQLite database setup configuration.load_configuration(configuration_filename) assert configuration.database_configuration.sqlite is not None + # Seed random for reproducible benchmark data + random.seed(42) + # we need to start each benchmark with empty database configuration.database_configuration.sqlite.db_path = str(tmp_path / "database.db")
294-298: Redundant assertion:len() >= 0is always true.The assertion on line 298 is a tautology—
len()always returns a non-negative integer. Consider removing it or replacing with a more meaningful check if needed.🧹 Proposed fix
user_conversations = query.all() assert user_conversations is not None - assert len(user_conversations) >= 0
316-321: Variableidshadows the builtin.Using
idas a loop variable (also at lines 393, 465) shadows Python's built-inid()function. While harmless here, consider usingrecord_idoridxfor clarity.🧹 Proposed fix
- for id in range(records_to_insert): - store_new_user_conversation(session, str(id)) + for record_id in range(records_to_insert): + store_new_user_conversation(session, str(record_id))
| # try to load the configuration containing SQLite database setup | ||
| configuration.load_configuration(configuration_filename) | ||
| assert configuration.database_configuration.sqlite is not None | ||
|
|
||
| # we need to start each benchmark with empty database | ||
| configuration.database_configuration.sqlite.db_path = str(tmp_path / "database.db") | ||
|
|
||
| # initialize database session and create tables | ||
| database.initialize_database() | ||
| database.create_tables() |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if pytest-xdist or parallel test configuration exists
echo "=== Checking for parallel test configuration in config files ==="
find . -type f \( -name "pytest.ini" -o -name "setup.cfg" -o -name "pyproject.toml" -o -name "tox.ini" -o -name "conftest.py" \) | head -20 | while read f; do
echo "--- File: $f ---"
if grep -q "xdist\|addopts.*-n\|parallel\|looponfail" "$f" 2>/dev/null; then
grep -n "xdist\|addopts.*-n\|parallel\|looponfail" "$f" || true
fi
done
echo ""
echo "=== Checking for pytest markers in benchmarks directory ==="
rg "pytest.mark" tests/benchmarks/ -A 2 || echo "No special markers found"
echo ""
echo "=== Examining the test file structure ==="
head -70 tests/benchmarks/test_app_database.py | tail -50Repository: lightspeed-core/lightspeed-stack
Length of output: 2161
The global configuration mutation pattern limits flexibility for future parallel execution.
The fixture mutates configuration.database_configuration.sqlite.db_path directly on a global object (line 57). While benchmarks don't currently run in parallel, this pattern prevents future parallelization without refactoring. The use of tmp_path correctly isolates database files per test, but modifying global configuration state is fragile.
Either add documentation explicitly stating benchmarks must run sequentially, or refactor the fixture to avoid mutating global state (e.g., using test-scoped configuration instances or dependency injection).
🤖 Prompt for AI Agents
In `@tests/benchmarks/test_app_database.py` around lines 52 - 61, The test mutates
global configuration.database_configuration.sqlite.db_path which is fragile for
parallel tests; instead, avoid changing global state by using a test-scoped
configuration or injection: create a local test config (or use monkeypatch) to
set sqlite.db_path from tmp_path and pass that into
database.initialize_database/create_tables (or refactor initialize_database to
accept a db_path parameter), ensuring you reference and update the test-local
copy rather than configuration.database_configuration.sqlite.db_path so each
test gets its own isolated DB without touching globals.
Description
LCORE-1249: stub for benchmarks
Type of change
Tools used to create PR
Related Tickets & Documents
Summary by CodeRabbit
Tests
Documentation
Chores
✏️ Tip: You can customize this high-level summary in your review settings.