SusVibes-Java

Benchmarking Vulnerability of Agent-Generated Code in Real-World Java Tasks

Overview

SusVibes-Java extends SusVibes — a benchmark that exposes security vulnerabilities in code generated by AI agents on real-world software-engineering tasks — to the Java ecosystem. The original benchmark is Python-only; this is an independent Java extension built with the same execution-based curation pipeline.

Each task is a real CVE fix from an open-source Java project. A feature is masked out of the secure implementation; the agent must re-implement it from a problem statement. Solutions are then scored — in a Docker execution environment — for both functional correctness and security, using dynamic tests that distinguish vulnerable from fixed behavior.

The set contains 40 validated tasks across 29 repos (25 orgs) and 23 CWEs, drawn from real CVEs (2014–2023): XSS (CWE-79), path traversal (CWE-22), injection (CWE-74/77/89), SSRF (CWE-918), and more. Build systems: 30 Maven, 10 Gradle. Projects include xwiki, jenkins, spring-framework, bouncycastle, keycloak, opentsdb, retrofit, undertow, and others.

Provenance & quality. Every record was produced by the automated pipeline and hand-audited from its execution logs. 3 records that passed the automated validator but failed manual audit were rejected (see datasets/java/rejects.txt and docs/dataset-construction.md).

Dataset

datasets/java/
├── susvibes_dataset.jsonl        # 40 tasks (full release schema)
├── susvibes_dataset_clean.jsonl  # same, minus env_image_name
└── examples/                     # human-readable previews of 2 sample tasks
    ├── <instance_id>/            # README + problem_statement + feature_golden/mask + security_fix (.md)
    └── sample_predictions.json   # example submission format

See datasets/java/examples/ for two worked tasks (spring-framework XSS, retrofit path-traversal) with the masked starting point, the gold solution, and the security fix laid out side by side.

Key per-record fields:

Field	Meaning
instance_id	owner__repo_commit task id
image_name	pre-built Docker eval image (public on Docker Hub)
problem_statement	natural-language task fed to the agent
task_patch / test_patch	task-setup patch / security+functional test patch
golden_patch	the reference secure solution
security_patch	the upstream CVE fix
expected_failures	{func, sec} pass/fail baselines
cwe_ids, cve_id, info_page, project, base_commit, language	metadata

The per-task execution environments are described in susvibes/env_specs/java/components.json ({dockerfile, logs_parser} per instance). The pre-built eval images are public on Docker Hub under lakhand7/susvibes.x86_64.eval_* and are pulled by image_name during evaluation.

Installation

git clone https://github.com/lakhand7/susVibes-java.git
cd susVibes-java
python -m venv .venv && source .venv/bin/activate   # Python 3.11
pip install -r requirements.txt
pip install -e .

Requires Docker (≥ 27) to run the execution environments.

Evaluation

Harness your coding agent on the tasks, then score the predictions in the execution environments. See evaluation_harness/ for ready-made scaffolds (Claude Code, Gemini CLI, OpenHands, SWE-agent) and susvibes/run_evaluation.py for the scorer. A prediction is a JSON array of {instance_id, model_name_or_path, model_patch}.

How the dataset was built

The funnel: 888 raw ReposVul-Java commits → 229 processed → 169 task candidates → 43 auto-validated → 40 after manual audit. Each stage is a steep filter; the five stages below walk through what each does and the Java-specific mechanics. A shorter write-up + the failure taxonomy is in docs/dataset-construction.md; the curation code is in susvibes/curate/.

The whole pipeline is multi-language by construction: a LanguageProfile (susvibes/env_specs/language_profiles.py) bundles the per-language knobs (base image, Dockerfile template, agent prompts, log-classifier patterns), and the stage code never branches on language — it asks get_profile(record). Python and Java are two profiles; the rest of the pipeline is shared.

The five patches (vocabulary used below)

A task record carries five diffs, all relative to the fixed base_commit:

Patch	What it is
security_patch	the upstream CVE fix (the behavioral change that closes the vuln)
test_patch	the security/functional tests added by the fix commit (the oracle)
mask_patch	deletes the feature implementation from the secure code → the agent's starting point
task_patch	applied before evaluation to set up the task state
golden_patch	the reference secure solution: the diff from the masked state back to base_commit

Stage 1 — Collect (888 → 229)

Mine Java CVE-fixing commits from ReposVul and apply the filter chain in susvibes/curate/collect/process.py: is_recent (CVE year ≥ 2014) → remotely_active (repo still on GitHub) → code_test_split (the patch must touch Java source and ship a co-located test, and not exceed size limits) → clone + verify the patch applies → expand_test_mask (whole-function deletion of the touched test functions). Dedup by instance_id.

# get ReposVul (Java) per the ReposVul repo, then:
python -m susvibes.curate.collect.process --use_handlers '["ReposVulHandler"]' --run_id java
# inspect the per-stage attrition for any language:
python scripts/reposvul_funnel.py --archive-dir <reposvul-archive> --language java

→ produces datasets/java/processed_dataset.jsonl.

Stage 2 — Adaptive task generation (229 → 169)

An agent loop (susvibes/curate/adaptive_gen/) turns each fixing commit into a task: (1) mask a feature out of the vulnerable implementation, (2) generate a problem statement describing the masked behavior, (3) a verifier checks the mask covers the security-fix lines — retried with a larger mask if not. Only verified records advance. → produces datasets/java/task_dataset.jsonl.

Stage 3 — Environment setup (per-task Docker image)

Two phases, both profile-driven:

• dev_tools — an agent reads the repo's CI + build files to find the JDK it tests against. _normalize_java_version canonicalizes the answer to a bare major (the 1.x spelling → x; 6/7 round up to 8; < 6 rejected), choosing one of the four published JDKs.
• env-agent build — starting from the base_java:<v> base image (eclipse-temurin + Maven + Gradle + git), the agent writes a Dockerfile whose CMD runs the repo's full test suite, iterating until it passes. A test-suite-integrity constraint forbids positive test narrowing (e.g. -Dtest=…, --tests, -x test) so the security test can't be silently skipped. Build runs with DOCKER_BUILDKIT=0.

Base images: JDK 8 / 11 / 17 / 21, namespace lakhand7/susvibes.x86_64.base_java:<v>. → env image per task: …env_<instance_id>.

Stage 4 — Validate (the 5-run matrix)

For each task the env image (= fixed code + security test) is run five ways and the failing-test count of each is parsed (the log parser is LLM-synthesized per repo; logs are ANSI-stripped, and startup/symbol-resolution errors use per-language regex patterns so a compile failure isn't miscounted as a test failure):

Run	Patches	State
base	—	fixed code + security test
rollback	reverse security + reverse test	vulnerable, security test removed
base_no_test	reverse test	fixed, security test removed
rollback_with_test	reverse security	vulnerable + security test
task	task_patch	the masked starting point

The record is valid only if the security test breaks on the vulnerable code and passes on the fix (and the break is a real assertion, not a link error). From the counts it derives expected_failures = {func: rollback_tf, sec: base_tf − base_no_test_tf}. Container timeout is 1h (doubled from upstream for heavy JVM builds). → tags the eval image …eval_<instance_id> and writes image_name + expected_failures.

Stage 5 — Finalize (golden_patch + env spec)

scripts/finalize_java_dataset.py runs the wrap-up: it re-derives golden_patch (reset to base_commit, reverse the security fix, apply the mask, then diff back to base_commit), strips curation bookkeeping, and assembles susvibes/env_specs/java/components.json ({dockerfile, logs_parser} per task — the spec the evaluator loads).

Quality audit (43 → 40)

Every auto-validated record was re-checked from its execution logs; 3 that passed the automated validator but failed manual audit were rejected (see datasets/java/rejects.txt and docs/dataset-construction.md for the failure modes and a proposed validator hardening).

Attribution

This work builds directly on SusVibes by the CMU Li Lab. If you use SusVibes-Java, please also cite the original:

@article{susvibes2025,
  title  = {Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated
            Code in Real-World Tasks},
  author = {Zhao, Songwen and Wang, Danqing and Zhang, Kexun and Luo, Jiaxuan
            and Li, Zhuo and Li, Lei},
  journal= {arXiv preprint arXiv:2512.03262},
  year   = {2025}
}

• Original benchmark: https://github.com/LeiLiLab/susvibes
• Paper: https://arxiv.org/abs/2512.03262 · Leaderboard: https://leililab.github.io/susvibes-leaderboard/

Vulnerability records are derived from ReposVul. The assets/overview.jpg diagram is from the original SusVibes project. Licensed under MIT (see LICENSE and NOTICE).