Benchmarking Self-Hosted LLMs for Offensive Security

TrustedSec ran 4,800 evaluation runs across six self-hosted models (gemma4:31b, qwen3.5:27b, devstral-small-2:24b, nemotron-3-super, qwen3-coder and qwen3:32b) against eight OWASP Juice Shop challenges, with gemma4 topping the leaderboard at a 98.5% pass rate using just an HTTP request function and basic encoding helpers. The catch: while local models reliably crush single-step exploitation like SQL injection auth bypass and IDOR, multi-step data extraction via UNION-based and blind SQL injection went so badly (qwen3.5:27b scored 0 of 100 on UNION extraction) that those challenges were dropped. TrustedSec reads this as a gap between knowing a payload class exists and applying it, though its deliberately minimal harness with no memory handling may have contributed. Deng et al. reached a similar conclusion after evaluating five LLM pentest agents: missing tools and weak prompts can be engineered away, but planning and state-management failures persist across models, with agents exhausting their context before finishing an attack chain. Your local gemma can probably pop a login form, but it still can’t plan a heist.