Skillgrade and the moment AI agents' 'skills' start needing unit tests

By GitHub · July 10, 2026.

Minko Gechev, an engineer well known in Google's Angular ecosystem, has released Skillgrade, an open-source tool (MIT license, already with several hundred stars) that solves a very specific problem: how to know whether an AI agent actually discovers and correctly uses an 'Agent Skill' —the SKILL.md file and associated resources that Claude Code, Gemini CLI, Codex, or other agents load to specialize in a task—. Until now, checking that a skill works was a manual and not very rigorous exercise: you wrote the instruction, tested it by hand with whichever agent you were using, and trusted that it would keep working the same way next time.

Skillgrade turns that into a reproducible process. It defines tasks in an eval.yaml file, runs them in a sandbox (Docker by default, or local for CI environments) against the chosen agent —Gemini, Claude, Codex, OpenCode, or any custom CLI via a generic adapter—, and scores the result with two types of 'graders': a deterministic one, which runs a script and checks objective facts (does the file exist? is the content correct?), and another based on an LLM judge that evaluates the session transcript against a qualitative rubric (did it follow the expected workflow? was it efficient?). Both scores are combined with configurable weights into a final grade, and the tool offers presets —smoke, reliable, regression— depending on how many attempts (trials) you want to run to estimate the success rate. It also integrates into CI pipelines, failing the build if the success rate drops below a threshold.

The contextual detail matters as much as the tool itself: barely one point and zero comments on Hacker News. It is not a high-impact story, and it shouldn't be inflated. But it is exactly the kind of early signal worth tracking, because it points to an underlying trend we had already been noting at Zendoric: as agents stop being demos and become production components —with skills, plugins, MCP, permissions, and token budgets—, a layer of reliability engineering is emerging around them that did not exist before. First came spend and permission control (agent FinOps); now comes testing. It is the same pattern that conventional software went through twenty years ago: when a component becomes critical, it is no longer enough for it to 'work on my machine' and it starts requiring test suites, reproducible sandboxes, and continuous integration.

Our reading is that this is good news, albeit modest. 'Skills' are consolidating as the packaging unit through which agents acquire specific capabilities —a kind of plugin for LLMs—, and a plugin ecosystem with no way to verify that they keep working after each model update is a fragile ecosystem. Tools like Skillgrade reduce that risk: they allow detecting regressions when the underlying model changes version, comparing which agent runs a specific skill better, and replacing blind faith in the prompt with measurable evidence. It is the same philosophy we apply in our own quality indices versus vendors' marketing: measure, don't take things for granted.

That said, the structural limitation should not be lost sight of: part of the grading system depends on another LLM acting as judge, which pushes the reliability problem one level up —who audits the evaluator?—. It is a known limit of the entire evals industry, not a flaw exclusive to this tool, and it will probably keep being addressed through deterministic graders where possible and stricter rubrics where not. In the short term, tools like this will remain niche, used by teams that are already building agents seriously; in the medium term, if the adoption of skills as a standard format keeps growing —as it seems to be—, this testing layer will stop being optional. And that is, ultimately, the kind of unglamorous work —sandboxes, rubrics, CI— that separates an agent ecosystem that shows promise from one that can truly be trusted.