Previous Post: Chronicles of Development Struggles in Building a Local LLM MCP Web Search Feature
The ordeal began with reports that two posts automatically uploaded at 7 AM contained nearly identical content, prompting fixes to the duplicate posting logic, implementation of automated verification to catch mixed Korean-English texts, and unfortunately, amidst these challenges, the Hermes agent experienced issues with SQLite locking, rendering it unresponsive for several days.
Duplicate Post Incident and Automation Halt
The investigation began with a report of identical posts on both Korean and English blogs despite a user reportedly disabling automatic posting at 08:00 AM. Checking the `dailypipeline2` job scheduled to run daily at 07:00 AM via launchd revealed it had already completed execution and deregistered it using `launchctl bootout`. Further investigation uncovered a more significant revelation: the Hermes skill documentation (version new3) explicitly stated on July 29, 2026, "This automated 07:00 posting will be discontinued and replaced with a news workflow requiring manual approval." Essentially, while the decision had been made, the launchd job remained active, causing the discrepancy.
Redesigning Duplicate Prevention Logic
The previous `11_dedup.py` script aggregated four key indicators (CPI, interest rates, unemployment rate, GDP) into a single string for exact matching comparisons, leading to an entire entry being flagged as "changed" if even minor rounding or revision discrepancies occurred, essentially rewriting identical content unnecessarily. This has been revised to perform individual field comparisons with tolerance for numerical errors, complemented by the addition of `check_topic_history` to compare similarity with recent headlines. This enhanced logic is now integrated into the news publishing script (`14_write_news_article.py`), ensuring automatic application across five skills (new3, pub3, kor-news, us-news, global-news), with 273 comprehensive tests confirming successful implementation.
Discovery of mixed Korean-English text and implementation of automated verification added
A user reported that a KR blog post featured an English title with interspersed English paragraphs within the text body. Upon direct querying via the Blogger API, our local model disregarded the instruction to write in Korean, generating several initial paragraphs in English instead. The affected post was rewritten entirely in Korean using `posts().update()`, followed by a comprehensive scan of all 46 posts (KR 17 + EN 29) at a paragraph-level Korean/English ratio to ensure no further instances existed. To prevent recurrence, a function `language_mix_violation()` was implemented to validate language ratios before posting, triggering a re-generation if violated, followed by mandatory Korean draft processing if necessary. This solution was validated on a KOSPI semiconductor topic using Ollama.
The Hermes Agent Has Become Completely Non-Functional
The next day, executing `/kor-news` resulted in a complete failure with the error message "session storage could not be written." Upon reviewing logs, approximately 300 instances of `sqlite3.OperationalError: database is locked` were found. The root cause was a WAL reset corruption bug in SQLite 3.51.0 used by Hermes, forcing the application to employ an older DELETE journal method. Concurrent access by the gateway process and desktop app background servers to the same database state led to repeated locking conflicts. Updating Hermes with `hermes update` to SQLite version 3.53.1 resolved the underlying issue.
The model fabricates progress without invoking external tools.
After fixing both SQLite and ddgs and running `/us-news`, reports indicated a lack of response. Upon manually inspecting the session database, the model responded twice with messages like "Ready to process," "Collection complete, moving to the next step," yet there were no actual tool executions (tool_turns=0). This wasn't due to failed searches but rather a scenario where attempts were not made at all, merely proceeding verbally. With infrastructure issues resolved, the issue was deemed a reliability concern in executing local model instructions, recommending a retry along with pre-heating Ollama beforehand.
Trouble Shooting
| Symptoms | Cause | Resolution |
|---|---|---|
| Automatic posting continues despite instructions to stop at 07:00. | Skill document decision made, but launchd job not actually released. | Unregister using `launchctl bootout` |
| Despite minimal changes in indicators, duplicate content continues to be generated. | Combine four indicators into one string for exact matching comparison | Comparison by Fields + Allowable Numerical Error |
| An English paragraph is mixed into a Korean blog post. | Local model ignores Korean instructions and generates some paragraphs in English. | Post-language ratio check, retry if violated; fail, enforce initial draft. |
| Immediately after `hermes update`, web searches on `/kor-news` ceased entirely. | Missing existing `ddgs` package in the new Python runtime for SQLite modifications. | Reinstall `pip install ddgs` in the new venv |
| Appears unresponsive after executing `/us-news` | The model only says "I will collect" without actually calling any tools (tool_turns=0). | Confirmation of no bug, recommend retry and preheat Ollama beforehand. |
Reflections / Next Steps
The key takeaway from this session was the stark difference between documenting a decision to "pause" in documentation and actually implementing that decision at the system level. While code bugs can be caught through testing, this "gap between decision and execution" often requires digging through logs to uncover. Additionally, fixing infrastructure inadvertently caused ripple effects (e.g., fixing SQLite led to ddgs disappearing), reinforcing the necessity to rigorously test core functionalities after major updates. Moving forward, I aim to develop a more structured approach to prevent local models from fabricating progress reports without invoking necessary tool calls.
Comments
Post a Comment