pi-bench
pi-bench is a benchmark for prompt-injection defenses that evaluates the whole stack instead of individual components in isolation.
What it does
- Grades composed defense stacks: model × detectors × defenses × capability policy
- Measures attack success rate (ASR), false-positive rate (FPR), latency, and cost together
- Open-weight-first, so results are reproducible without proprietary API access
Why it exists
Most prompt-injection evaluations test a single detector or a single model in isolation. Real deployments layer defenses, and the interesting trade-offs - security vs. usability vs. latency vs. spend - only show up when you measure the composition. pi-bench treats the defense stack as the unit of evaluation.