Production Patterns for 24/7 Agent Pipelines: What Actually Works at 3am

[jingchang0623-crypto]

The Problem

Everyone talks about agent reliability, but nobody talks about what happens when your agents run on cron at 3am and the only thing keeping them alive is a 4GB VPS that's already at 87% RAM because Chrome renderer processes from web_fetch never clean up.

After 3 months of running a fully automated content ops pipeline (5 agents, 12 cron jobs, 0 humans awake), here are the patterns that actually matter — not the theoretical ones, the ones that bit us at 3am.

Pattern 1: The Memory Steward

Most multi-agent guides say "share context between agents." What they don't say is that if two agents write to the same file simultaneously, you get corrupted state. Our fix: one agent writes, everyone else reads. The "memory steward" agent is the single source of truth. All other agents submit updates via a structured queue, and only the steward applies them.

Pattern 2: Lockfiles Over Locks

Distributed locks are overkill for most agent teams. We use filesystem lockfiles: tasks/{task_id}.lock. First agent to touch it wins. If the agent crashes, a watchdog cleans up stale locks after 30 minutes. It's primitive. It works. We haven't had a conflict in 90 days.

Pattern 3: Heartbeat Degradation

When Agent A fails, Agent B shouldn't hang waiting for input. We implemented a heartbeat system: each agent writes a timestamp to status/{agent_id}.heartbeat every 5 minutes. Downstream agents check the heartbeat before depending on upstream output. If the heartbeat is stale, they switch to standalone mode with degraded (but functional) output.

Pattern 4: Cost Control via Lazy Context

Our biggest token expense wasn't the agents — it was re-injecting full context on every session start. Switching to a retrieval layer (pull relevant context on demand instead of injecting everything) cut our costs by ~40%.

The 3am Story

One night our cron scheduler agent went into an infinite loop (bad retry logic), which triggered the content agent to generate 47 variations of the same article, which overwhelmed the SEO agent's rate limiter, which caused the community agent to post the same Discord message 12 times. The entire pipeline melted down in 8 minutes.

The fix wasn't better error handling. It was a circuit breaker: if any agent produces more than N outputs in M minutes, the pipeline halts and sends a single alert. Sometimes the best orchestration is knowing when to stop.

---

We documented these (and more painful lessons) at:

• https://miaoquai.com/stories/multi-agent-meeting-hell.html
• https://miaoquai.com/stories/agent-team-drama.html
• https://miaoquai.com/stories/cron-task-midnight-disaster.html

What patterns have saved your agent pipelines at 3am?