postmortem - MetalTorque Intelligence

postmortem

When a 401 Error Became Valid Data: How Poisoned Rows Survived Two Days Undetected

Summary On April 8, 2026, our research analysis service lost access to its primary AI provider when a Groq API key expired. Instead of failing loudly, the service treated the provider's 401 Invalid API Key error string as a valid analysis result — and wrote it into the output

postmortem

How a Privilege Boundary Turned a Single 401 Into 205 Consecutive Failures

At Ledd Consulting, we run 25 microservices and 7 AI agents on a single VPS. We built a session reliability watcher that detects agent failures, applies exponential backoff, and auto-restarts degraded services. On April 3rd, a single expired authentication token on one agent exposed a blind spot we hadn'

postmortem

When Two Users on One VPS Corrupted Our Shared State — and How We Hardened 25 Services in a Weekend

Summary On March 25, 2026, we added a second system user to our single-VPS infrastructure running 25 AI microservices at Ledd Consulting. Within hours, the event bus started silently dropping events, the automation state database threw SQLITE_BUSY errors under load, and two JSON state files were truncated to zero

postmortem

When Our Notification System Said "Sent" But Nobody Got the Email

Summary On March 28, 2026, we discovered that our notification router and event bus had been silently dropping operator emails for approximately 36 hours. The root cause: we trusted the exit code of our mail CLI tool (himalaya) as proof of delivery. The tool returned 0 (success) even when the

postmortem

When Our Drip Sequence Engine Sent 3 Weeks of Emails in One Morning

At Ledd Consulting we run 25 services and 60+ scheduled timers on a single VPS. Most of the time, that architecture is a strength — low cost, low latency, easy to reason about. But on the morning of February 18th, it turned into a liability. A routine VPS restart caused our

postmortem

How a Social Monitor Storm Created 1,400 Duplicate Events in 20 Minutes

At Ledd Consulting, we run social monitoring agents that scan developer communities for signals — new articles, forum questions, hiring posts. These monitors feed into our event bus, which fans out to notification services, an AI watcher for opportunity scoring, and a lead pipeline. On March 8th, a routine tag expansion

cost-management

When Our Cost Tracker Caught a $900 Overnight API Spend — and How We Auto-Capped It

At Ledd Consulting, we run 25 microservices and 7 autonomous agents on a single VPS. Our daily API bill is typically under $2. One Tuesday morning, it was $900. Here is exactly what happened, how we caught it, and the auto-capping system we built so it stays impossible for this