I added "duplicate content prevention" logic to the economic blog automation pipeline. I expected an easy finish since all the mock tests passed, but running it against the real API and the real local model turned up one bug after another — from one that nearly disabled the whole logic to ones that broke the HTML output.
Why duplication prevention was needed
FRED/ECOS indicators only update monthly (CPI) or quarterly (GDP), but the pipeline auto-generates four combinations — US/Korea × English/Korean — every day at 7 a.m. On days the data hasn't changed, the posts end up practically identical, which is bad for SEO and AdSense review. I'd already decided to "compare against the last run's values and either skip or substitute alternate content," but hadn't written the code yet.
Implementing it in 11_dedup.py
I built a new module that saves and compares the previous indicator snapshot per topic×language combination. A DEDUP_MODE setting in .env toggles three modes — off always generates, skip skips a combination that hasn't changed, and substitute swaps the market-recap post for a concept-explainer post about what the indicator even means instead. I passed five scenarios with every external call mocked out, and updated the docs.
Snapshot contamination the moment I ran it against the real API
Running the same topic with only the language changed produced different snapshots. It turned out the GDP value was being stored with its language-specific unit label attached ("USD Trillion" vs. "조 달러"). Left as-is, this would have judged the data as "changed" every time just because the language differed, even with identical underlying data — effectively disabling duplication prevention. FRED's accompanying revision-timestamp metadata was also getting mixed in, so I fixed it to store only date and value, and the raw pre-conversion value at that.
Running it through real Ollama broke the HTML
Generating a concept-explainer post produced malformed nesting like <p><h2>title</h2>body</p>. When the model wrote only the h2 and skipped the p, the existing fallback wrapped the whole block in a p tag anyway — the same bug that was already lurking in the market-recap parser. I added a function that splits blocks by heading, converts markdown lists to ul/ol, and converts bold markup to strong. Running it again, the model this time wrapped the HTML heading in markdown bold on top of it, and after fixing that, a variant mixing markdown headings and bold showed up too. Watching a new variant break three times in a row reminded me, again, that you can never fully trust a small local model's output formatting.
Korean generation, another one — meta description leak
Running it in Korean too, the length-instruction text showed up tacked onto the end of the meta description. The existing filter only caught a "quantifier + unit" pattern like "about 850 characters," so it missed a newer variant that came with a "Length:" label attached. I split the rule so that a labeled instance gets caught even without a unit, while an unlabeled one still requires a unit to match — otherwise it would strip out legitimate parenthetical text too. While I was in there, I also caught an off-by-one bug where the function meant to cap the meta description at 150 characters was actually emitting 151.
Auto-starting Ollama + automating token refresh
Ollama wasn't in the login items, so if I didn't manually launch it after turning on the MacBook, the whole 7 a.m. scheduled run could fail. I added a function that checks the server's status at pipeline start and launches it directly if it's off. The Instagram long-lived token also expires every 60 days with no refresh automation in place, so I built and registered a new launchd job that runs on a 30-day cycle (the first registration attempt threw an I/O error from conflicting with an already-loaded job; unloading and re-registering fixed it).
Trouble Shooting
| Symptom | Cause | Fix |
|---|---|---|
| Snapshot differs just from changing the language | The GDP snapshot mixed in a language-specific unit label and FRED metadata | Store only date and value, using the raw pre-conversion value |
| Malformed p/h2 nesting in the body | When the model wrote only h2, the fallback wrapped the whole block in p | Added a block-level HTML normalization function |
| Leftover **<h2>..</h2>** style markup | The model wrapped the HTML heading in markdown bold again | Added a regex that strips the bold wrapper off the tag first |
| "(Length: about 1950 characters)" left in the meta description | The leak filter didn't anticipate the newer labeled variant | Split the rule by whether a label is present (allow omitting the unit when a label exists) |
| Risk of every scheduled run failing if Ollama isn't running | Ollama isn't registered as a login item | Added a preflight step where the pipeline wakes the server itself |
Reflections / Next steps
The biggest lesson today: mock tests only prove the logic is correct — they say nothing about the real API's actual response shape or a local model's actual output. The snapshot-contamination bug was one mocking alone would never have caught. Next session I need to confirm drafts are actually being saved to Blogger, and that the skip logic really kicks in a day or two later.
Comments
Post a Comment