01 / The story
It started with 78 installed skills
The first question was practical. Should we keep a skill, rewrite its description, merge it with another or remove it? A folder full of instructions cannot answer that. We needed to see what each skill said, when the agent used it and whether another instruction already covered the same task.
We began with one local skill tree. The dashboard made it possible to compare our reading of the files with actual usage records. WhichSkills grew out of the next question: could we apply a consistent method to a larger public sample and let other people check the results?
02 / The problem
An installed skill is not necessarily a useful skill
Usage counts miss context. A skill may cover an occasional task, overlap with another skill or have a description that never triggers. A repository's popularity does not explain any of those things.
Safety checks have a similar limitation. The same command can be an instruction to execute, a documented example or a test fixture. We needed a way to separate a pattern match from a conclusion, and keep enough evidence for someone else to challenge that conclusion.
03 / How it works
Connect the files, judgments and review
The dashboard loads skill files and plugin manifests, checks local transcripts for evidence of use and compares skills for overlap. Jev adds judgments about usefulness, redundancy and clarity. Its recommendation stays separate from the decision we record. No recorded invocation is a reason to investigate, not proof that a skill has no value.
Take the same questions to a larger sample
For the published 20 September 2026 snapshot, we searched GitHub by queries and topics, excluded forks and selected the top 200 repositories by stars. Of those, 157 contained skills or plugin manifests. The loader found 18,041 skills and 557 manifests.
We recorded the source commits so a finding could point to the file we actually read. Stars helped define the sample; they did not decide which skills were useful or safe. This is a dated sample, not a complete inventory of everything available.
Use cheap checks first, then read the context
The static pass checked accompanying text files against 14 classes of patterns, including external data transfers, hidden instructions, credential access and automatic hooks. We deliberately made this pass sensitive. A deployment command or a security lesson can trigger it without being an attack.
Selected flags put a row into the review queue. We deduplicated the bodies before asking Jev to read them, so repeated instructions did not need repeated judgments. The published run contains 545 unique bodies read by Jev, not 18,041 model evaluations.
For each body, Jev answered two fixed questions: how risky the instructions appeared on a defined zero-to-three scale, and which kind of risk best described them. It returned distributions rather than a free-form verdict. We then opened every body scoring at least 1.5 at its pinned commit and recorded what we found.
The run recorded roughly 4.2 million input tokens and 93 seconds for the Jev pass. Those are measurements from this batch, not a promise about future runs. The published method includes the questions, queue rules and data files.
The most useful findings needed a human reading
One highly ranked file was a deliberately poisoned test fixture in NVIDIA's SkillSpector repository. Other flags came from offensive-security teaching material. The text contained real attack examples, but its purpose was testing or instruction. Treating every match as malware would have misrepresented the files.
A more relevant finding was a plugin hook that told the agent to act without asking the user. Jev identified this instruction from the full text even though our static rules did not have a specific category for that behaviour.
Our review did not identify malware aimed at the installing user in the reviewed material. That is not a safety certificate. The process did not inspect binaries, images or code fetched later at runtime, and the site records oversized inputs the API rejected.
Duplication was widespread too. The site reports 36% identical copies under its comparison method, which removes frontmatter and collapses whitespace. Much of the copying was the same owner publishing in more than one place. The provenance view distinguishes those mirrors from third-party copies and labels uncertain origins as the earliest known copy in the sample.
Keep the evidence beside the recommendation
Question wording mattered. A broad request to recommend an action tended towards keeping skills. More explicit criteria separated the results better. Context mattered too: a short file made entirely of an attack example could score higher than a long document quoting the same example.
That is why the dashboard keeps the underlying evidence and distributions visible. We want to inspect a recommendation, disagree with it and save a reason. The public site follows the same principle with commit-linked files, review notes and downloadable data. TypeSafe did not commission or fund the work; we used its public API.
Find additions that fit the work
Once we knew what was installed, we could ask what was missing. The Upgrade view builds a profile from local instructions, skills and usage, narrows the public index to candidates and asks Jev to read them against that profile. It ranks fit while accounting for the chance that an existing skill already does the same job.
Recent opening prompts are an optional input. If included, that text goes to Jev. The dashboard runs locally, but model judgments send their supplied context to TypeSafe. Check that context before evaluating private skills or work records.
The result is a tool for reviewing what we have and choosing additions deliberately. You can explore the published findings and searchable index before running anything yourself.
04 / Installation
Run it on your own skills
The dashboard needs Python 3 and uses the standard library with SQLite. There is no frontend build or package installation step. Clone the repository and start the local server:
git clone https://github.com/48Nauts-Operator/skill-dash
cd skill-dash
python3 server.py --port 3345Open http://localhost:3345. Loading files and running static checks do not need a model key. To enable Jev judgments, set TYPESAFE_API_KEY in the server environment before starting it. API access and usage charges are separate.
The guides below walk through setup, collection, review and sharing. They also provide a pinned code revision for following the recorded workflow.
