Bobbin: a 100% local AI pentesting agent that also publishes its failures

By Zendoric · September 7, 2026.

Bobbin is a coding agent designed for small models — 7 to 32 billion parameters, the size that runs on a single consumer GPU — executed locally through Ollama, the tool that makes it possible to run language models on your own computer without going through the cloud. According to its documentation, the project does not depend on any external Python package: standard library only. Its loop is simple: the model issues a structured tool call, the program executes it, the result comes back as a message and the model gets another turn, until it answers in prose or the step budget runs out.

The author compares Bobbin with aider, one of the reference coding agents, under the same test harness, whose raw results — 450 runs — are published in the repository itself. On its own multi-file refactoring fixtures, Bobbin wins comfortably: 107 successes out of 114 against 56 and 52 for aider's two configurations. But when it moves to real code — the pallets/click project, with 12,674 lines and 1,991 tests — the result depends on the model: with the 30-billion-parameter qwen3-coder it loses (6 out of 18 versus aider's 9); with nemotron it wins by a landslide (12 out of 18 versus 3). The author himself sums up the aggregate — 50% against 33% — as "a tie with a wide margin, not an advantage," a rare piece of honesty in AI tool marketing.

The project's underlying argument, however, is not to compete with aider: it is that running locally enables a category of work that a cloud model cannot touch for confidentiality reasons. Code under NDA, an audit for a client with a no-data-leakage clause, a pentest whose target cannot be described to an outside vendor: all of it can still be reviewed with a model that never leaves the machine. According to the project's own tests, in code review the three models evaluated (qwen3-coder:30b, nemotron-3.5-lightning and qwen2.5-coder:7b) detect the planted vulnerabilities without exception — SQL injection, command injection, weak hashing, embedded secret, predictable token, path traversal; the real cost is false positives, such as flagging a properly parameterized `subprocess.run()` call as insecure.

In active exploitation against an offline test target, the pattern flips: finding the flaw is the hard part. Only SQL injection falls reliably, and only with the two 30-billion-parameter models; IDOR (access to other users' resources through a failed ownership check) and data exposure via path brute-forcing fail across all models until a specific, narrowly scoped identifier-sweeping tool is added — not a generic shell — which does unlock IDOR. By design, Bobbin does not execute arbitrary commands: it generates a proof of concept that a human must run. The project also documents, in the same detail, the mechanisms that did not work — warnings, blocks and loop guards against a model that refuses to admit it found nothing — leaving them published as negative results instead of hiding them.

Our reading is that Bobbin's interest lies not in whether it beats or loses to aider, but in what it confirms about where AI-assisted offensive security is heading: the capability already exists for bounded tasks — finding a SQL injection, reviewing a file for known patterns — and is still missing for those that demand persistence and judgment, such as systematic enumeration. It is the same pattern we have seen in other contexts: cybersecurity works as a reliable thermometer of how far a model really goes, far more than any saturated benchmark. And the fact that it is a local model, running on a desktop GPU, that sustains that first audit filter is further proof that the open, locally executable frontier is closing gaps that a year ago only frontier models in the cloud could cover: the democratization underpinning our broader thesis about abundance comes not only from the big labs, but also from single-developer projects willing to measure and publish where they fail.

It is worth putting it at its real scale, though: this is an early-stage repository, with its own metrics unaudited by third parties and barely any traction in the community. Its value is not that of a launch that changes the market, but that of a display of methodology — publishing the 4,133 execution logs, failures included — that ought to be the norm in a sector that tends to sell capability with cherry-picked benchmarks. If anything deserves applause here, it is not that a small model matches a frontier one, but that someone bothers to say exactly where it does not.