I built a Hermes agent skill that takes a photo sent over Telegram and marks hazards or inspection points with numbered labels. I started by digging into how accurately a local vision model could ground bounding boxes, and ended up hitting two walls I didn't see coming.
The Coordinate Experiment: It Wasn't Pixels, It Was Normalization
First I measured bounding-box accuracy on a synthetic shapes image with my local brain model (qwen35-hermes). Asking "the image is 800x600, answer in pixel coordinates" gave an IoU of 0.00–0.09 — essentially random — and the model even returned coordinates outside the image bounds. Before swapping models, I recalled hearing that gemma4:31b worked fine at the office, so I suspected the prompt first instead. Asking again with coordinates normalized to 0–1000 on both axes jumped the IoU to 0.85–0.89. It turned out the Qwen-family VLM was trained to ground in normalized coordinates from the start. I confirmed this held up with a real office photo (finding a mouse) and a 6-object simultaneous numbering test before locking in the approach.
Two Limits That Showed Up in Real-World Testing
The multi-object numbering test surfaced two limitations. Adjacent similar objects — three monitors sitting side by side — got boxes that slightly overlapped, and when two identical Red Bull cans were in frame, the model only caught one unless I explicitly said "number each instance separately, don't merge them." I baked both of these into the skill's prompt rules.
The First Skill Build, and the Setback
For requests that need a documented rationale — safety inspections under Korea's Industrial Safety and Health Act, equipment checks — I added a "reasoning mode" (risk_level and reasoning fields) and shipped the skill under the name "image." Then, testing it for real on Telegram, typing "/image" returned Unknown command. It turned out Hermes already has a CLI-only built-in command named `/image` (attach a local file), so the name collided — and I also learned personal skills aren't registered in Telegram's official slash-command menu at all.
Switching to a natural-language request instead of the slash command, the photo still never came back. Digging through the logs, the real cause was different: the model never once called `skill_view` (the tool that actually reads the skill file) — it just kept calling `vision_analyze` and eventually wrapped up with a plain-text answer. A 9B-class local model simply couldn't reliably assemble the multi-step "analyze → extract JSON → save file → run script" procedure on its own.
The Redesign: Collapse Everything Into One Script
The fix was to minimize how much the model had to decide on its own. I merged the direct Ollama API call, JSON validation, coordinate conversion, and image annotation into a single script, so the model only needs to make one `terminal` call. I also renamed the skill to "photo-mark" to avoid the core-command collision. Re-testing on the office photo, it correctly caught all 5 items — including both Red Bull cans — in 118 seconds.
More Detail, and a Hanzi Contamination Bug
The user pointed out that the explanations felt thinner than before. It wasn't a context-window limit — the JSON schema had explicitly asked for a "one-line explanation." Loosening the reasoning field to "write 2–4 detailed sentences" produced much richer output, but in the process I noticed the occasional Chinese character slipping into the Korean text (things like 略显). That looks like a side effect of the underlying model's Chinese-language base.
Trouble Shooting
| Symptom | Cause | Fix |
|---|---|---|
| Bounding-box coordinates were essentially random (IoU 0.05), sometimes outside the image bounds | Asked for pixel coordinates — the model was still computing on a 0–1000 internal scale and reporting it as pixels regardless | Switched the prompt to 0–1000 normalized coordinates → IoU 0.85–0.89 |
| Typing "/image" on Telegram returned Unknown command | Name collision with Hermes's built-in CLI-only `/image` command (attach a local file) | Renamed the skill to "photo-mark" and switched to natural-language triggers |
| The photo never came back even with a natural-language request (twice in a row) | The model never called `skill_view` — it just kept calling `vision_analyze` and finished with a prose answer, unable to assemble the multi-step procedure on its own | Collapsed the whole procedure into a single script invoked with one `terminal` call |
| After adding the hanzi-contamination filter, responses started taking up to 8 minutes | The filter could regenerate up to 3 times, but the SKILL.md's terminal timeout was set too tight at 200s — the first attempt was force-killed and the model retried on top of that | Cut max retries from 3 to 2 and raised the timeout from 200 to 420 |
Takeaways / What's Next
The biggest lesson here was to question the prompt format before concluding "the model just can't do this," and to stop trusting a small local model to orchestrate multiple tool calls on its own — collapsing the procedure into a script is the more reliable path. Next up: attaching a fixed, internally-documented inspection standard (our proprietary 8-system-plus-foreign-material checklist) as reference material the skill can pull from.
Comments
Post a Comment