Ask two specific questions
The safety preset requests a risk score on a 0–3 scale and the dominant risk kind. Jev receives the full skill body, accompanying text, and static flags.
- Risk
- A score, confidence, and probability distribution.
- Kind
- Injection, exfiltration, destructive behavior, credentials, supply chain, obfuscation, self-modification, or none.
- Evidence
- Source locations, flags, model, question version, and token usage.
Inspect the first batch
Open jev-risk.json. Read several records against their source files before expanding the run. A record with an error field is a failed judgment, not a low-risk result.
Continue the run
Remove --limit 10 to process the remaining queue. The script saves results incrementally and skips already-recorded keys.
Saved errors are also recorded keys. Investigate their cause and handle those records before expecting a rerun to judge them again.
It’s working if
- Successful rows include risk, probabilities, and source locations.
- Usage is recorded alongside each result.
- You can distinguish failed judgments from completed ones.
Read the implementation
The queue runner defines the saved record and resume behavior. Compare with the published judgment data.