Self-Healing Homelab with AI Agents
Project Overview
This is for my own homelab: two servers, a NAS, and a firewall running about 100 containers. Several of the projects on this site run there.
A dedicated low-power server whose only job is operations. It runs a Claude Code session I can hand infrastructure work to from my phone. It also runs a set of scheduled jobs that handle routine maintenance on their own.
Scheduled AI jobs all go through one small job runner. It makes sure two runs can't overlap, stops a run that goes too long, and keeps a log of each one. The same setup handles the weekly Brush Quest releases. The agent makes one change, it gets built, and the main server deploys it, checks that the site still works, and rolls it back if it doesn't.
Built up through 2026, mostly in September and October. Most pieces were added after something actually broke.
An Intel NUC on my home network. It reaches the other machines over SSH and doesn't publish anything to the internet itself.
My main server stays busy with media transcoding, so moving scheduled and AI work off of it helped both. The other reason was outages. After a power outage the NAS takes the longest to boot, so the main server would come up with its network shares and Docker broken until I fixed them by hand. Now a scheduled check catches that and recovers it.
Giving an AI agent access to production systems also meant writing down rules it has to follow, like which services it can never restart remotely. I keep runbooks in Notion so each session starts from what's already known.
What It Handles
- After a power outage, it remounts the NAS share and starts Docker once the NAS is back. I can pause it during maintenance.
- It restarts a media processing service if it stops, and clears stuck items out of download queues on a schedule.
- It checks that the GPU driver still matches the kernel, so a kernel update no longer takes down the services that need the GPU.
- Grafana alerts go to Telegram. I tuned them so short blips don't send an alert.
- It runs the weekly Brush Quest releases, each with a health check and automatic rollback.
- It reboots the cable modem once a week, since it slows down after a few weeks up.
Still open: it only covers the failures I've run into so far. I add to it when something new breaks.