A premium, observability-enabled evaluation dashboard and interactive playground to compare Open Source (OSS) and Frontier LLMs across safety, factual accuracy, and hallucination rate paradigms.
📊 Try the Main Evaluation Dashboard Here - https://ai-assistant-eval-g7wrhuzeaom8p5zunuxgpd.streamlit.app
(Hosted on Streamlit Community Cloud)
🦙 Try the Qwen OSS Assistant Here - https://huggingface.co/spaces/VD09/qwen-assistant
(Hosted on Hugging Face Spaces)
"Compare Open Source vs Frontier models side-by-side"
"Evaluate outputs with an impartial LLM-as-a-Judge"
"Monitor and block unsafe requests with Dual-Layer Guardrails"
Python 3.10+
Streamlit (Interactive Dashboard)
Groq API (Llama 3.3 70B & 3.1 8B Classifier)
Gemini API (Gemini 3.5 & 2.5 Flash)
SQLite (Observability Logging)
Plotly (Visualizations)
🦙 Open Source Models (OSS)
🦙 Llama 3.3 70B (Default)
🤖 Qwen 3 32B (Reasoning)
⚡ Frontier Models
⚡ Gemini 3.5 Flash (SOTA)
⚡ Gemini 2.5 Flash (Default)
1. Clone the repo
git clone https://github.com/YOUR_USERNAME/ai-assistant-eval
cd ai-assistant-eval2. Install dependencies
pip install -r requirements.txt3. Configure Environment Variables
Create a .env file in the root directory:
GROQ_API_KEY="your-groq-api-key-here"
GEMINI_API_KEY="your-gemini-api-key-here"4. Run the Application
streamlit run app.py| Feature | Description |
|---|---|
| Interactive Chat Arena | Parallel multi-turn messaging panel for OSS and Frontier assistants. |
| Dual-Layer Guardrails | Uses llama-3.1-8b-instant to block toxic/jailbreak inputs and outputs. |
| LLM-as-a-Judge Evaluator | Impartial grading (1-10) with reasoning by Llama 3.3 70B. |
| System Observability | Real-time logging of prompts, guardrails, and latency via local SQLite. |
| Benchmark Manager | Create, edit, export/import prompt evaluation suites in JSON/CSV. |
| Component | Choice | Trade-Off |
|---|---|---|
| Guardrails | llama-3.1-8b-instant with zero-shot classification |
Extra 150-250ms latency vs. local embedding/regex, but allows complex rule evaluations. |
| Judge Model | Llama 3.3 70B (Groq) |
Near-frontier level reasoning with higher rate limits and speed than free-tier API models. |
| Logging | Local SQLite (observability.db) |
Zero-config and zero-cost local setup, but limits horizontal scalability compared to external observability platforms. |
| OSS Model | Deployment Target | Est. Latency | Est. Cost |
|---|---|---|---|
| Qwen2.5-72B-Instruct | Hugging Face Spaces (Serverless API) | ~0.8s - 1.2s / request | $0.00 (Free Tier) |
| Llama-3.1-8B (Alternative) | Groq API (Cloud) | ~0.3s - 0.5s / request | Free Tier ($0 for eval) |
| Qwen2.5-7B-Instruct (Alternative) | Hugging Face Dedicated Endpoint (T4 GPU) | ~1.5s - 2.5s / request | ~$0.60 / hour |
1. Guardrail Quantization: Port custom safety classification models directly into local ONNX runtime instances inside the container to eliminate the API network hop completely.
2. Multi-Judge Consensus: Implement a voting consensus protocol (e.g. Llama-70B, GPT-4o-Mini, and Gemini-Flash) to average out judge bias and handle outlier scores.
3. Conversational Drift Observability: Log embedding vectors of conversation history to detect user topic drift and predict potential jailbreak sequences before they trigger a violation.
