VectorCheck is a regression testing framework designed for AI agents and LLM-based applications.

Traditional string comparison methods (assert a == b) are often insufficient for testing generative AI, where responses can vary with each execution. VectorCheck leverages Vector Similarity and Semantic Evaluation to validate tests based on "Intent", ensuring that semantically identical outputs pass even if the exact wording differs.

• 🧠 Semantic Testing: Validates test results based on meaning rather than exact text matching. (Uses LLM Judge or Embedding Similarity)
• ⏪ Golden Data Replay: Fetches verified "Golden Data" (production logs) from Weaviate and replays them to ensure regression stability.
• ⚡ Zero Config Setup: Automatically generates optimal default configurations (including .env) if they are missing.
• 🔍 Smart Auto-Discovery: Automatically detects and registers target functions within your project.
• 📊 CLI Dashboard: Provides real-time, intuitive Pass/Fail reports directly in your terminal.

You can easily install VectorCheck via pip. (Requires the VectorWave framework)

pip install vectorcheck

Run the following command in your project's root directory:

vw test

On the first run, it will automatically generate a .env file and apply local environment settings.

You can define different testing strategies for each function. Use similarity for functions with high randomness, and exact for deterministic functions.

[vectorcheck]
python_paths = test_ex  # Path to the module being tested

; 1. LLM Summary Function (High Randomness) -> Pass if similarity > 85%
[test:test_ex.example.generate_review_summary]
strategy = similarity
threshold = 0.85

; 2. Payment Process (Partial Changes like IDs) -> Pass if similarity > 95%
[test:test_ex.example.process_payment]
strategy = similarity
threshold = 0.95

; 3. URL Generation (Deterministic) -> Must match exactly
[test:test_ex.example.generate_report]
strategy = exact

Choose between HuggingFace (Free, Local) or OpenAI (Paid, High Performance) for your vectorizer.

# Default: Use Local HuggingFace Model (Zero Cost)
VECTORIZER="huggingface"
HF_MODEL_NAME="sentence-transformers/all-MiniLM-L6-v2"

# Optional: Use OpenAI
# OPENAI_API_KEY=sk-...

# Run all tests (follows ini configuration)
vw test

# Run a specific target function
vw test --target test_ex.example.process_payment

# Force Semantic (LLM Judge) mode for all tests
vw test --semantic

# Force Similarity mode (Threshold > 0.8) for all tests
vw test --threshold 0.8

Export execution logs from the DB for training or analysis purposes.

vw export --output ./data/dataset.jsonl

VectorCheck operates in integration with the VectorWave core.

1. Trace: Captures function Inputs/Outputs and metadata, storing them in Weaviate (Vector DB).
2. Fetch: Retrieves verified 'Golden Data' during testing.
3. Replay: Re-executes the current code logic using the same inputs.
4. Evaluate: Compares the past results (Expected) with the current results (Actual) in vector space.

Distributed under the MIT License. See LICENSE for more information.

Developed by Kim Junyeong.