How I Built a Full Status Alert System

TL;DR by Goose▾

Ingredients

• Headless Linux server — the old laptop from the previous two posts, running 24/7 (already set up)
• cron — Linux task scheduler, built-in (free)
• Resend — email API for all alert delivery (free tier: 3,000 emails/month)
• Claude Code — terminal AI for writing every script ($200/yr)
• healthchecks.io — dead-man’s switch that alerts when a scheduled job stops running (free tier)
• Supabase project — for the Numerator game database health check (already set up)

The Problem: A Server You Can’t See

A headless server is quiet by design. That’s the point — it runs in the background, lid closed, in another room. But quiet also means invisible. Without monitoring, if your site goes down at 2am, you won’t know until someone tells you. If your database has an outage and a contact form stops saving submissions, you’ll find out whenever you next check by hand. If a scheduled job silently fails, you get no email and no clue.

The solution isn’t to check things manually — that defeats the purpose of automation. The solution is to make the system tell you when something is wrong. Every important job should either succeed quietly or fail loudly. Here’s the full stack:

• Alert 1: Website Uptime Monitor — checks that joseandgoose.com is responding
• Alert 2: Garmin Recap Failure Check — dead-man’s switch if the morning recap doesn’t run
• Alert 3: Daily Fail2ban Report — how many new attackers were turned away
• Alert 4: Supabase Health + GitHub Activity — weekly database check and code activity
• Alert 5: Personal Server Changelog — Claude writes a plain-English weekly standup
• Alert 6: Weekly Status Report Email — everything in one weekly digest
• The Meta-Alert: healthchecks.io — alerts if the server itself goes offline

Alert 1: Website Uptime Monitor

Every few minutes

The most basic question: is joseandgoose.com responding? A curl request every few minutes, checked against an expected HTTP status code. If it returns anything other than 200, send an alert.

🔧 Developer section: Uptime monitor script

• curl -s -o /dev/null -w "%{http_code}" https://joseandgoose.com — gets the HTTP status code silently
• If status ≠ 200: sends a Resend email with the status code, timestamp, and a note to check Vercel logs
• If status = 200: logs the timestamp quietly, no email
• Also logs to a daily CSV so I can spot patterns (slow responses during peak hours, etc.)

In the first month it caught two blips: a Vercel deployment that briefly returned a 503 during a cold start, and a broken build. Both arrived as an email before anyone else noticed.

Alert 2: Garmin Recap Failure Check

Every morning

A while after the Garmin recap is scheduled, a recap file should exist for today. If it doesn’t, something broke overnight — and I should know before I’ve been waiting all day for a recap email that’s never coming.

🔧 Developer section: Garmin failure check script

• Runs daily, after the recap’s scheduled time
• Checks whether today’s recap file exists
• If it exists: silent pass, no email
• If it’s missing: sends an alert email with the last 20 lines of the recap log
• Alert subject: "⚠️ Garmin Recap Failed — [DATE]"

This is a dead-man’s switch pattern: instead of the job alerting on success, a second job alerts on missing success. It catches silent failures — crashes, auth errors, network timeouts — that don’t generate their own error output.

Alert 3: Daily Fail2ban Report

Once a day

Fail2ban bans IPs automatically, but I wanted a daily snapshot: how many new IPs got banned today? Is that number trending up (could indicate a targeted scan) or holding steady (normal background noise)?

🔧 Developer section: Fail2ban report script

• sudo fail2ban-client status sshd — outputs total banned count
• Script reads the current total, compares to yesterday’s saved count
• Calculates: new bans = current total − previous total
• Saves today’s count to a small state file for tomorrow
• Sends an email: "🛡️ Fail2ban Daily Report — X new bans today (Y total)"

The delta matters more than the total. A large cumulative count after weeks of running is expected. An unusual spike in a single day is worth investigating.

Alert 4: Supabase Health + GitHub Activity (Weekly)

Weekly

Two separate checks that share a slot because they’re both weekly sanity checks rather than urgent alerts:

🔧 Developer section: Supabase health check

• Runs a simple read query against the Supabase REST API
• If the API returns a valid response: logs the count, no email
• If it errors (503, timeout, auth failure): sends an alert with the error body
• Also confirms the game database is live
• Credentials load from a local env file, never from the script itself

🔧 Developer section: GitHub activity report

• Calls the GitHub API for my recent public activity
• Filters for the past 7 days of events: pushes, PRs, issues, stars
• Formats into a short summary and includes it in the weekly report email
• Uses a read-only token

Alert 5: Personal Server Changelog (Weekly)

Weekly — Claude-generated

Once a week, Claude writes a short narrative of what the server did that week. It’s not a metrics dump — it’s a 3–5 sentence changelog in plain English, like a standup from the server to me.

🔧 Developer section: Claude-generated changelog

• Script collects stats: Garmin recaps generated this week, new Fail2ban bans, AI jobs completed, uptime log entries, disk usage, site downtime events
• Passes stats to claude -p "..." with a prompt asking for a casual, 3–5 sentence changelog
• Claude output is saved to a temp file and folded into the weekly report
• Runs before the weekly report email so it’s ready in time

A recent example output from Claude:

Alert 6: Weekly Status Report Email

Weekly — the full picture

All the pieces come together in one weekly email: changelog, Supabase health, GitHub activity, Fail2ban weekly total, disk space, and a resource summary. It’s the one email that tells me everything about the past week without opening a terminal.

🔧 Developer section: Weekly report assembly

• Runs after the other weekly jobs have completed
• Reads the Claude changelog from its temp file
• Reads the Supabase and GitHub outputs
• Pulls resource stats: df -h for disk, last CSV entry from resources-YYYY-MM-DD.csv
• Assembles into an HTML email via Resend API
• Subject: "🖥️ Server Weekly — [WEEK OF DATE]"

The Meta-Alert: healthchecks.io

There’s one failure mode none of the above covers: what if the server itself goes down? A crashed machine runs no cron and sends no email, so it can’t report its own outage. The solution is a dead-man’s switch hosted externally.

🔧 Developer section: healthchecks.io heartbeat

• Free account at healthchecks.io creates a unique URL — the "check"
• If the URL isn’t pinged within a set interval, healthchecks.io sends a failure alert
• Added to cron on a regular interval: curl -s https://hc-ping.com/[uuid] sends a heartbeat
• If the heartbeats stop, healthchecks.io emails me

Final Output

The server now manages itself. I never log in to check if things are running. I get emails when something is wrong, and I get a weekly report that tells me everything is fine. The no-email state is the good state.

• Site downtime → email
• Garmin recap failure → email
• AI job failures → email immediately
• Remote SSH login → instant email
• Server offline → alert from outside, via healthchecks.io
• Everything working fine → one weekly summary email

What went fast

• Each individual script — Claude Code wrote every bash script from a plain-English description. Uptime monitor: 15 minutes. Fail2ban report: 20 minutes. Each one is simple; the value comes from having all of them running together.
• Resend API reuse — same API key, same sender, same pattern for every email. Once the first alert email worked, every subsequent one took 5 minutes to wire up.
• Cron scheduling — crontab -e, paste a line, save. Linux cron is reliable and dead-simple for time-based jobs.

What needed patience

• Alert fatigue tuning — initial versions sent too many emails. Had to add cooldowns, state files (to remember last-alerted time), and delta calculations (ban count delta, not total). Getting the signal-to-noise ratio right took iteration.
• Cron environment — cron jobs run in a minimal shell environment without your normal PATH. Scripts that work fine in a terminal can silently fail in cron because python3, node, or claude can’t be found. Fix: explicitly set PATH at the top of every cron command or in the crontab header.
• Weekly job ordering — the weekly report depends on outputs from earlier jobs. If any upstream job is slow, the report reads an empty file. Added fallback messages ("data unavailable") for each section so the report always sends even if one input is missing.
• healthchecks.io setup — the concept clicked immediately; finding the right "grace period" setting (how long to wait before alerting) took some tuning. Too short and a brief network hiccup triggers a false alarm. Set it a little longer than your heartbeat interval.

The hardest part of a home server isn’t setting it up — it’s knowing what’s happening on it without babysitting it. This alert stack is the answer. Every meaningful event surfaces as an email. Everything else is silence, which is good news.