What it is
BASTUS (Batch Automation for Safety Testing and Usability Scenarios) is an
automated red-teaming suite. An uncensored attacker LLM runs many parallel,
multi-turn (and multimodal) conversations against a target LLM + harness, trying
to break its safety guardrails — then reports what got through. The goal is to catch
child-safety, health/legal, crime/violence, sexual-content, and deepfake/NCII failures
before they reach production, including ones you didn't know to look for.
How it works
- Attacker → Target → Judge loop. The attacker escalates over multiple turns
using the Crescendo method — start benign, then escalate by referencing the
target's own prior replies.
- Multi-judge scoring. Each target response is scored 0–100 for how
specific and useful any elicited harmful content is — not merely whether the
model refused (avoiding "empty" jailbreaks).
- Beam search. Your Agents setting is the number of parallel conversation
branches per goal; the highest-scoring branches survive each turn.
- Everything is recorded. Every run gets a numeric ID, all conversations are
saved to Postgres, and turns stream live into this console.
Open-source lineage
BASTUS is an original implementation whose architecture follows established open
red-teaming work:
- Microsoft PyRIT — the attacker / target / scoring / orchestration pattern.
- Crescendo (Microsoft Research) — the multi-turn escalation method.
- MLCommons hazard taxonomy / Meta Llama Guard — the S1–S14 disallowed-content
categories used as the run's settable parameters.
- StrongREJECT — the "score usefulness, not just refusal" judging principle.
- HarmBench — standardized red-team evaluation practice.
The attacker is an abliterated (refusal-removed) open model — default
huihui-ai/Qwen3-32B-abliterated — self-hosted on RunPod via vLLM, so
adversarial probes are generated on infrastructure you control.
How to test a target LLM
- Provision the attacker server (top-left). Wait for the RunPod status to read
Ready. No run can launch until then.
- Point BASTUS at your target. Set
TARGET_ENDPOINT,
TARGET_API_KEY, and TARGET_MODEL (any OpenAI-compatible
endpoint) in the server configuration.
- Configure the run: choose disallowed categories (S1–S14), the number of tests
(goals), agents (parallel branches), branching, max turns, and the break threshold.
- Launch and watch conversations stream. Pause, resume, or abort at any time.
- Review results: per-category Attack Success Rate plus full transcripts. Leave
Mock (offline) checked to demo against a simulated target with no real calls.
Safety boundaries
- Child safety (S4) is tested text-refusal only — never with images.
- Deepfake / NCII (S12) uses benign seed images; a "break" means the target's
willingness or partial compliance, never a finished explicit artifact.