To gain insight into what my Telegram-based autonomous agent Hermes is currently doing, I integrated the open-source "Pixel Agents" project, which visually represents code execution through pixel art characters resembling Claude, onto Hermes as well. This process involved forking the project to develop a new provider, ultimately addressing underlying structural limitations.
Step 1: Investigate Pixel Agents
Upon investigation, Pixel Agents appeared to be a VS Code extension/standalone server that visualized terminal "Claude" code sessions as pixel office characters. The documentation described it as "provider-agnostic," though the actual implementation supported only the "Claude Code." The architecture involved posting session events to a local HTTP hook endpoint (/api/hooks/<provider>), resulting in character appearance; similarly, Hermes could send events in the same format.
Step 2: Implement New Hermes Provider
Cloned the repository to inspect the HookProvider interface and, referencing the Claude provider, rewrote hermes.ts. Given Hermes' inherent hook system located at ~/.hermes/hooks/, I developed additional Python handlers alongside to emit events to Pixel Agents servers at key points such as session initiation, tool usage, and turn completion.
Third Stage: Hidden Single Provider Issue
Initially, I thought registration would be straightforward, but upon closer inspection during runtime (AgentRuntime), only one provider was statically bound at initialization, disregarding the origin of events. The system consistently interpreted inputs in the manner of Claude, irrespective of the actual provider. By modifying handleEvent to dynamically identify the correct provider each time, we achieved seamless concurrent operation between the Claude codebase and Hermes.
Step 4: Deployment for Real-World Use
The temporary workspace folder disappears after the session, so I moved everything permanently to ~/pixel-agents and created start/stop scripts on the desktop. Installing the Hermes hook file and restarting the gateway initiated the reflection of Telegram conversations on the office screen.
Step 5: Persistent Floating Window on Screen
In response to requests for a persistent window rather than browser tabs, I developed a lightweight wrapper using Electron. This always-on-top frameless window reuses existing server functionality without additional server logic, integrating seamlessly. Subsequent improvements addressed issues such as repeated resets of zoom/position, overly large Layout/Settings buttons in a small window, and limited mouse vertical scrolling for directional movement (addressed with Shift+scroll functionality).
Step 6: Replace with Community Layout
Given the basic office space was limited to two rooms, I opted for a simpler setup and sourced the "Severance Office" layout from the community gallery "Pixel Index," featuring a single, wide room. Since the standalone server's layout file update functionality was exclusive to VS Code, I replaced the file and restarted the server to apply the changes.
Trouble Shooting
| Symptoms | Cause | Solution method |
|---|---|---|
| Hermes events are processed only in the Claude style label/format. | Runtime uses only one fixed provider at creation time, ignoring providerId per event. | Patches implemented by temporarily replacing the correct provider for each event. |
| End script failed to locate extraneous processes. | When executed in the background using `node ... | tee log &`, $! refers to the PID of `tee`, not `node`. | Modify to capture the actual PID of the node using process substitution (`> > (tee ...)`) |
| Hermes character not appearing again after server restart | Hermes sends location information only in the "first message" of a session; information lost when the server restarts if it was only stored in memory. | Update to retransmit location data before every event (safely ignored for known sessions) |
| Character nametags appear as "Coding" and cannot distinguish between Claude and Hermes. | The displayed value was actually a shortened name of the project folder, not the provider name. | Redeploy "providerLabel" across all server and webview sections, adding "Claude Code"/"Hermes" tags above characters. |
| Detaching and attaching an external monitor hides floating windows from view. | Saved window position is based on the previous monitor arrangement, now extending beyond the screen boundaries. | Reset stored position values and restart at default location (bottom right of main screen). |
Reflections / Next Steps
Relying on the "provider-agnostic" claim in open-source documentation rather than thoroughly tracing the actual code flow proved most valuable in this project. This approach enabled the creation of genuine multi-provider support at the runtime level, rather than superficial feature additions. Future enhancements may include integrating a desktop pet with floating characters on a transparent background or automating window repositioning upon detecting changes in external monitors.
Comments
Post a Comment