When the 3am job fails

You have a job that pulls the day's Stripe payouts into a spreadsheet at 3am. It runs fine for two months. Then one Tuesday it doesn't, and you find out the following Monday, when someone asks why the numbers stop at the 4th.

Nothing told you. No error in your inbox, no red dot anywhere you'd look. The script threw, the scheduler shrugged, and the only signal was the absence of output. If you've ever searched for how to know when a cron job fails, you probably have your own version of that Monday.

Agent automations make this worse, not better. You ask your agent to set up a nightly digest or a weekly cleanup, it writes the code, schedules it, and the conversation ends. Nobody is watching it now. Not you, and not the agent, which has moved on to the next chat.

So when I built jobs into Minion, I spent as much time on the failure side as on the scheduling side.

Why cron jobs fail silently

The classic setup has no memory. Cron runs a command, the command exits, and whatever it printed goes to a log file or a local mailbox nobody reads. The failure exists for a second and then it's gone.

The causes are boring, which is part of why they go unnoticed:

• An upstream API starts returning 401 because a key was rotated.
• A dependency bumps a version and changes a return shape.
• Someone edits the job's settings and drops a parameter.
• A secret expires and nothing was watching the date.

None of those are hard to fix. They're hard to notice. By the time the missing output tells you, you've lost days, and the logs that would have explained it are often gone too.

Dedicated cron job monitoring tools solve part of this with heartbeats: your job pings a URL when it finishes, and you get alerted when the ping stops. That's a good pattern for crontabs on your own servers. What I wanted for Minion was different. The jobs run inside Minion, so Minion already knows when one fails. The question was what to do with that.

How to know when a cron job fails: keep every run

Every execute call, job, webhook, event handler and app request in Minion becomes a run. Each run keeps its console logs, its returned value, how long it took, and on failure the error and stack. Secret values are redacted.

The first question I want answered in the morning is "is anything broken?", so that's the first call:

const summary = await minion.runs.summary()
return summary.errors

That counts open failures over the last 24 hours, with a breakdown by source, so you can see whether it's a job, a webhook or something your agent ran by hand. Then the list, and then the one that matters:

const { runs } = await minion.runs.list({ errorTriage: 'open', limit: 10 })
const run = await minion.runs.get({ id: runs[0].id })
return { error: run.error, logs: run.logs }

Runs are retained for 32 days on the free plan and 90 on paid. Logs and results are kept for 7 days on free and 30 on paid. That's long enough that "it broke last Tuesday" is still something you can look at, not something you reconstruct from memory.

Minion writes the diagnosis note for you

A stack trace tells you where something broke. It rarely tells you why it broke today when it worked yesterday. That's the part that eats a morning, diffing things by hand.

So when a run fails, Minion writes a note. It has the likely cause, the last good run, and what changed since then: package versions, code, job settings, secrets, and the upstream HTTP status if a fetch went wrong. If the same failure keeps happening, the repeats are grouped into one note instead of fifty. Every diagnosis note is tagged run-diagnosis:

const { notes } = await minion.notes.search({ tags: ['run-diagnosis'] })

This is the bit that changes the morning. Instead of "the payout job failed", your agent opens a note that says it started failing after a specific package version, or right after a secret was changed, or when the Stripe call began returning 401. Most of the time that's the whole investigation.

Triage, then a fix that waits for you

Not every failure deserves attention. Some are known noise: a flaky upstream that recovers on the next run. Some you've already fixed. Leaving them in the list means the real ones get buried, which is how you end up ignoring the list.

So each failed run can be marked open, resolved or ignored, with a note:

await minion.runs.update({ id: run.id, errorTriage: 'resolved', note: 'fixed in v4' })

Triaged runs drop out of the open count and the default list. The error and logs stay, so the history is still there if it comes back. Any member can triage, and the run records who did it and when.

When your agent has a fix, it can propose one without shipping it. minion.runs.proposeFix takes the failed run, the package version the fix is based on, and the complete fixed files, and saves them as a new unpublished draft of the package:

const pkg = await minion.packages.get({ id: run.packageId })
await minion.runs.proposeFix({
  id: run.id,
  baseVersionId: pkg.versions[0].id,
  files: fixedFiles,
})

The draft can't run until a person reviews and publishes that exact version. I wanted that line to be firm. An agent fixing its own 3am job and redeploying it unattended is exactly the kind of thing that turns one quiet failure into two.

The waiting list: what only a person can clear

Some failures aren't code at all. An integration needs reconnecting. A secret expires in five days. A host needs approving before a secret can be sent to it. Mail is sitting in quarantine. A new job has never had a green run, so nobody knows if it works yet.

An agent can't do any of those. It can only tell you. The useless version of telling you is "check the dashboard". So Minion keeps a waiting list : everything in the organization that's blocked on a person, each with an actionUrl that goes straight to the page where it gets cleared, and a who field that says who can act ("you", "any member", "owners and admins").

const { count, items, waitingUrl } = await minion.waiting.list()
return items.map((i) => `${i.title} (${i.who}): ${i.actionUrl}`)

Failed runs show up there too until someone triages them, and so do failing jobs. It's computed from live state, so items vanish as soon as they're cleared. When I ask my agent "what needs me?", that's the call it makes, and I get a short list of links instead of a tour of the settings pages.

How do you monitor AI agent automations?

To be clear about what this is: Minion doesn't text you or post to Slack when a run fails. Out of the box, failures show up in runs, diagnosis notes and the waiting list, where you or your agent find them.

If you do want a push, it's one package away. Minion emits run.error.recorded when any run fails or times out, and job.failed when a job does. A saved package can subscribe to either one, and its handler is an ordinary package run, so it can email the team or post to Slack. For Slack, store the incoming webhook's path as a secret, call https://hooks.slack.com/services/{{secret:slackWebhookPath}}, and approve hooks.slack.com for that secret. Subscribe to one of the two topics, not both, or a failing job notifies you twice. The package subscriptions docs have the manifest shape and filters like { "source": "job" }.

What I'd actually suggest for monitoring AI agent automations is simpler than alerting. Make "is anything broken?" the first thing your agent checks when you start a session. runs.summary(), then waiting.list(). It takes a second, and it means a 3am failure shows up the next morning instead of the next Monday. The Activity docs cover runs end to end, and the Waiting docs cover the list.

It's the same idea as reminders that admit when they fail : the quiet failure is the one that hurts, so the work goes into making it loud.

Try it

Connect your agent and ask it: "Did any of my runs fail in the last day, and is anything waiting on me?" If something did fail, ask it to read the diagnosis note before it touches the code.

Start at first steps , or sign up and connect your first agent.