How We Built a Morning Briefing That Synthesizes Overnight Activity From 25 Services Into One Email
When you run 25 microservices, 7 autonomous agents, and 60+ scheduled timers on a single VPS, mornings become a problem. We wake up to a system that has been busy — scraping job listings, running research pipelines, extracting actions, building knowledge bases, checking for hallucinations — all between 1 AM and 7 AM. Every service writes its own logs, its own reports, its own JSON files.
For months, we started each day the same way: SSH in, check five directories, open three dashboards, scan two email threads, read a Telegram backlog. Forty minutes of context-loading before doing any real work. At Ledd Consulting, we decided this was an engineering problem worth solving properly.
The Problem
The challenge seems simple on the surface: read a bunch of files and send an email. The reality is messier.
Our overnight pipeline runs in stages. Research runs at 1 AM. Action extraction runs at 2 AM. Cross-agent synthesis runs at 2:15 AM. Knowledge base updates at 2:30 AM. A hallucination checker runs at 7:10 AM UTC. Each stage writes output to different directories in different formats — JSON reports, Markdown briefs, plain text action lists. Some stages depend on others. Some fail silently. Some produce output that only matters if another stage also produced output.
We needed a system that could:
- Aggregate temporal activity across a 6-hour overnight window
- Normalize heterogeneous output formats (JSON, Markdown, plain text)
- Rank and prioritize — surface the 5 things that matter, suppress the 50 that are routine
- Close the loop — feed the morning briefing into the evening reflection, which feeds back into the next morning
Every "daily digest" tool we evaluated assumed a single data source. Ours has ten.
Architecture Overview
The system is two scripts that form a feedback loop:
┌─────────────────────────────────────────────────────┐
│ OVERNIGHT PIPELINE │
│ 1:00 AM Research agents run (5 parallel) │
│ 1:30 AM Conversational agents run (2 sequential) │
│ 2:00 AM Action extractor processes all output │
│ 2:15 AM Cross-agent synthesizer builds brief │
│ 2:30 AM Knowledge base update │
│ 3:00 AM Builder agents execute on extracted tasks │
└──────────────────────┬──────────────────────────────┘
│ writes files to disk
▼
┌──────────────────────────────────────────────────────┐
│ MORNING BRIEFING (7:00 AM EST) │
│ Reads: job-reports/, property-reports/, briefs/, │
│ actions/, reports/, proposals/, trackers │
│ Queries: event-trigger stats, marketplace revenue, │
│ Ghost CMS, system health │
│ Output: Single consolidated email + working memory │
└──────────────────────┬───────────────────────────────┘
│ sets context for the day
▼
┌──────────────────────────────────────────────────────┐
│ EVENING REFLECTION (10:00 PM EST) │
│ Reads: daily plan, memory notes, working memory, │
│ event stats, bid logs, timer health │
│ Output: Summary email + updated working memory │
│ (carried forward → next morning briefing) │
└──────────────────────────────────────────────────────┘
Both scripts are plain Node.js. The morning briefing is a pure aggregator — it reads files, queries a few HTTP endpoints, formats everything, and sends one email. The evening reflection is an LLM-powered summarizer — it gathers the day's data and asks Claude to produce a structured wrap-up. Together, they form a 24-hour awareness cycle.
Implementation Walkthrough
Temporal File Discovery
The core challenge is finding "what happened overnight" across directories where files accumulate over weeks. We use a simple pattern: every overnight process writes files with date-prefixed names. The briefing finds the latest one:
function findLatestFile(dir, prefix, ext) {
try {
const files = fs.readdirSync(dir)
.filter(f => f.startsWith(prefix) && f.endsWith(ext))
.sort()
.reverse();
return files.length > 0 ? path.join(dir, files[0]) : null;
} catch (e) {
return null;
}
}
This looks trivial. It is. And that's the point. We considered a database, a message queue, an event-driven approach where each service would publish its results to a central store. All of those would have been more "correct." All of them would have required every service to integrate with yet another dependency.
The filesystem-as-message-bus approach works because of one convention: every service writes {prefix}-{YYYY-MM-DD}.{ext}. Lexicographic sorting gives us temporal ordering for free. findLatestFile is called ten times across the briefing — once per data source. Each call is independent, each is fault-tolerant (returns null on any error), and the whole thing runs in under 50ms.
Cross-Source Aggregation With Graceful Degradation
The briefing consolidates ten distinct data sources. Here's how the job-matching section works — and the pattern repeats for every source:
function getJobsSummary() {
log('Reading job reports...');
const file = findLatestFile(JOB_REPORTS_DIR, 'scraper-', '.json');
if (!file) return { text: 'No job scraper data found.', count: 0, topJobs: [] };
const data = safeReadJSON(file);
if (!data) return { text: 'Could not parse job report.', count: 0, topJobs: [] };
const categorized = data.categorized || {};
const sideGigs = categorized.sideGigs || 0;
const partTime = categorized.partTime || 0;
const fullTime = categorized.fullTime || 0;
const total = data.newJobs || (sideGigs + partTime + fullTime);
const jobs = data.jobs || [];
const scored = jobs.map(j => {
const maxScore = (j.classifications || []).reduce(
(max, c) => Math.max(max, c.score || 0), 0
);
const category = (j.classifications || []).reduce(
(best, c) => (c.score || 0) > (best.score || 0) ? c : best, {}
).category || 'unknown';
return { ...j, maxScore, topCategory: category };
});
scored.sort((a, b) => b.maxScore - a.maxScore);
const topJobs = scored.slice(0, 5);
let text = `${total} new matches (${sideGigs} side gigs, ${partTime} part-time, ${fullTime} full-time)`;
text += `\n Scanned ${data.totalMatches || 0} listings across ${
(data.sources || []).filter(s => s.status === 'ok').length
} active sources`;
if (topJobs.length > 0) {
text += '\n Top matches:';
topJobs.forEach((j, i) => {
const salary = j.salary ? ` — ${j.salary}` : '';
text += `\n ${i + 1}. [${j.topCategory}] ${j.title}${salary}`;
text += `\n ${j.company || 'Unknown'} | ${j.source}`;
});
}
return { text, count: total, topJobs };
}
Every getter function follows the same contract: return { text, count, ...metadata }. If the source is missing, return a neutral message and zero counts. If the file exists but is corrupt, same thing. The briefing assembles all ten sections regardless of which ones succeeded. On a typical morning, 8–9 of 10 sources have fresh data. Occasionally one agent fails overnight. The briefing still sends — it just notes "No data found" for that section and moves on.
This is the single most important design decision in the whole system: every data source is optional. The briefing always sends. An incomplete briefing at 7 AM is infinitely more useful than a complete briefing that occasionally fails to send because one upstream service had a bad night.
The Briefing-Reflection Feedback Loop
The morning briefing sets context. The evening reflection closes the loop. Here's how the reflection gathers the day's data:
function gatherDayData() {
const data = {};
// Today's plan
data.plan = readSafe(PLAN_FILE);
// Today's memory notes
data.todayMemory = readSafe(path.join(MEMORY_DIR, `${today()}.md`));
// Current working memory
data.workingMemory = readSafe(WORKING_MEMORY);
// Check service health
try {
const health = execSync(
'systemctl list-units --type=timer --state=running --no-pager --plain 2>/dev/null | head -30',
{ encoding: 'utf8', timeout: 10000 }
);
data.runningTimers = health;
} catch { data.runningTimers = '(unable to check)'; }
// Event trigger stats
try {
const triggerStats = execSync(
'curl -s http://127.0.0.1:8080/stats 2>/dev/null',
{ encoding: 'utf8', timeout: 5000 }
);
data.eventStats = triggerStats;
} catch { data.eventStats = '{}'; }
return data;
}
The reflection reads the plan that was set in the morning, compares it against what actually happened (via memory notes, event stats, bid logs), and produces two outputs: a summary email and an updated working memory file.
That working memory file is the bridge. The evening reflection writes it at 10 PM. The overnight agents read it at 1 AM to inform their research priorities. The morning briefing reads it at 7 AM to show continuity. It's a 24-hour cognitive loop stored in a single Markdown file.
const prompt = `Review today's activity and create the evening wrap-up.
## Today's Plan
${data.plan || '(no plan found)'}
## Today's Memory Notes
${data.todayMemory || '(none)'}
## Working Memory
${data.workingMemory || '(empty)'}
## Event Triggers Today
${data.eventStats}
Create TWO outputs:
### OUTPUT 1: Evening Summary (for email)
Brief, 5-10 lines max:
- What was accomplished today
- What failed or was skipped
- Key metrics (bids sent, events processed, content created)
- Any issues that need attention tomorrow
### OUTPUT 2: Updated Working Memory
...
Write both outputs separated by "---SPLIT---"`;
The ---SPLIT--- delimiter is intentionally low-tech. We tried structured JSON output from the LLM. It worked 90% of the time. The 10% where it added trailing commas or markdown-escaped quotes inside JSON strings caused silent failures. A plain text delimiter works 100% of the time because the LLM treats it as a document separator rather than a syntax element it might get creative with.
Delivery: Email as the Universal Interface
We use himalaya — a terminal email client — to send the briefing. When that fails, we fall back to our notification router:
try {
const emailCmd = `himalaya message send << 'EMAILEOF'
From: System <system@example.com>
To: ${EMAIL_TO}
Subject: [System] Evening Wrap-Up — ${today()}
${summary}
EMAILEOF`;
execSync(emailCmd, { encoding: 'utf8', timeout: 30000, shell: '/bin/bash' });
} catch {
// Fallback: notification router
try {
execSync(`curl -s -X POST http://127.0.0.1:5000/notify \
-H 'Content-Type: application/json' \
-d '${JSON.stringify({
source: 'evening-reflection',
priority: 'normal',
category: 'system',
title: 'Evening Wrap-Up',
message: summary.slice(0, 1000)
})}'`, { timeout: 10000 });
} catch {}
}
Email is the delivery mechanism because it's the one channel that is always available, works across every device, and leaves a searchable archive. We considered Telegram (already integrated), Slack, and a dashboard. The briefing still shows up on those channels via the notification router fallback. But email is primary because when we're scanning our phone at 7:03 AM, that's where we look.
What Surprised Us
Working memory drift was real. The evening reflection updates a working memory file that persists across days. After two weeks, we noticed the "Carried Forward" section growing unboundedly — tasks the LLM kept carrying forward because they were technically unfinished, even though they'd become irrelevant. We added a hard rule: carried-forward items older than 3 days get dropped. The LLM respects this consistently when instructed explicitly in the prompt.
The briefing became the system's heartbeat. We built it as a convenience. It became our primary monitoring tool. When the briefing arrives and three sections say "No data found," we know exactly which overnight stages failed — faster than checking dashboards. One morning, the jobs section, property section, and research section all showed empty. That meant the 1 AM stage had failed completely, which pointed to a Node.js memory issue we traced and fixed within 20 minutes.
Ten sections was the ceiling. We started with 6 sections, grew to 10, and briefly tried 13. At 13, we stopped reading the email thoroughly. We'd skim the first 4 sections and ignore the rest. Ten is the maximum for a digest that actually gets read end-to-end. We collapsed the three lowest-value sections into a single "Other Activity" line.
Lessons Learned
Convention over infrastructure. Date-prefixed filenames, standardized return types { text, count }, and directory-per-source organization eliminated the need for any coordination infrastructure. Every service just writes files. The briefing just reads files. A new agent joins the fleet by writing to a directory and adding 15 lines to the briefing script.
Always-send beats sometimes-perfect. The briefing has sent every single morning for 11 weeks straight. It has included partial data dozens of times. It has yet to fail to send. This reliability comes from treating every data source as optional and wrapping every I/O operation in try/catch that returns a neutral default.
Feedback loops need explicit decay. Any system where an LLM carries state forward will accumulate stale context. Build in explicit expiration. Our 3-day carry-forward limit keeps working memory at a useful size (~20 lines) instead of growing into an unreadable backlog.
Email is still the best briefing channel. Dashboards require you to go look. Chat notifications get buried. Email sits in an inbox until you process it, provides full-text search, and works on every device. For a once-daily summary, email remains the highest-signal delivery mechanism we've found.
LLM output parsing should be dumb. String split on a delimiter. Trim whitespace. Done. Structured output parsing from LLMs introduces fragility at exactly the moment you need reliability — when the system runs unattended at 10 PM and the results feed into tomorrow's pipeline.
Conclusion
The morning briefing took two days to build. It's 400 lines of plain Node.js with a single dependency (dotenv). It reads files, queries a few endpoints, formats text, and sends an email. There's nothing clever about it.
And yet it changed how we operate. Forty minutes of morning context-loading collapsed to a 90-second email scan. Overnight failures surface immediately instead of hiding until they compound. The feedback loop between morning briefing and evening reflection gives our agent fleet a memory that persists across the daily cycle.
The pattern generalizes to any team running autonomous agents: build an aggregation layer that treats every source as optional, delivers consistently, and closes the loop with a reflection step. The agents do the work overnight. The briefing tells you what they accomplished. The reflection tells them what to do next. Three scripts, one Markdown file, and zero additional infrastructure.
Need help building AI agent systems or designing multi-agent architectures? Ledd Consulting specializes in autonomous workflow design and agent orchestration for enterprise teams.