Your Runbook Is Rotting. Teach It to an Agent Instead.
Every experienced developer owns a graveyard of runbooks. The upgrade guide you wrote after the last painful framework migration. The deploy checklist in the team wiki. The "gotchas" doc that was accurate the week you wrote it. They all die the same death: the knowledge was real, the document was faithful, and then nobody — including you — reads it at the moment it matters.
I recently ran that cycle one more time, with a twist at the end. A multi-day engagement modernizing a Next.js codebase — the same rebuild whose foundation I documented in my stack post — produced a genuinely good playbook: measured facts, reproduced failures, hard-won ordering rules. And instead of filing it in the graveyard, I spent a day turning it into an agent skill — a package of instructions, reference material, and executable checks that an AI coding agent loads on demand and follows the next time the same class of work shows up. Then I tested whether it actually worked, with evaluations that compare an agent using the skill against an agent flying without it.
The short version of the results: the skilled agent scored 14/14 across the graded scenarios; the baseline agent scored 7/14 — and one of its failures was the exact mistake that had cost me a production-shaped outage in the original engagement. This post is the arc of how a playbook becomes a skill: what to capture, how to turn prose into gates, why the checks must be executable, and how to prove the whole thing works before you trust it. The specific skill automates Next.js modernization, but the method applies to any recurring engineering work you're tired of re-learning.
Step one: capture the playbook while the bruises are fresh
The raw material for a good skill is not documentation — it's engagement notes. During the modernization work, I kept a running playbook with a strict rule: every entry had to be something that actually happened, written down the day it happened. Not "be careful with caching" but "this specific tag-revalidation call went stale under these specific conditions, verified by this round-trip test."
That rule matters because the value of a playbook is concentrated in its specificity. Generic advice ("test after upgrading") is exactly what an AI agent — or a future you — already knows and already ignores. What neither of you knows is the symptom table: when a Payload CMS plugin configuration change silently requires a new database migration, the failure isn't a build error, it's a 500 at runtime, and the fix is running the migration-create command before anything ships. That entry exists in my playbook because I shipped the 500. The bruise is the documentation.
By the end of the engagement the playbook held a couple dozen of these: pitfalls with symptoms and enforced rules, an opinionated target stack, a phase ordering that had survived contact with reality, and a verification ladder — the sequence of checks, from typecheck to live smoke test, that constitutes "actually done" rather than "looks done." Notice what the playbook is not: it's not a rewrite of the official upgrade guides, which cover the framework's own breaking changes well. The playbook holds the layer the official docs can't: how this stack's pieces interact, which claims need proof in this deployment setup, and which failures announce themselves as something else entirely.
Which is where most stories end, and where the rot begins. A playbook is passive. It waits to be read, and reading is precisely what doesn't happen under deadline pressure — by humans or by agents barreling toward a quick fix.
Step two: understand what a skill actually is
An agent skill, in the sense Anthropic's tooling defines it, is refreshingly mundane: a folder containing a main instruction file, optional reference documents, and optional executable scripts, with a description that tells the agent when to load it. No fine-tuning, no retrieval infrastructure. When a task matches the description, the agent pulls the instructions into context and follows them; the reference files load only when needed, keeping the skill cheap until it's relevant.
The mundane packaging hides the important shift. A document describes; a skill prescribes. The playbook said "the upgrade went best in this order." The skill says: you are now in Pass 1, which is read-only inventory; you may not modify the repository until the plan produced in Pass 2 receives explicit human sign-off; each Apply phase must end with the checks green before the next phase opens. Same knowledge, different grammatical mood — and the mood is everything, because an agent mid-task doesn't consult wisdom. It follows procedure.
Turning my playbook into the skill meant three structural decisions, and they're the reusable part of this story.
Decision one: phases became gates
The playbook's phase ordering — inventory the codebase, plan the work, apply changes in dependency order, finish with verification — was advice. The skill turned it into four gated passes, where the gate is the point: the agent cannot proceed past certain boundaries without either green checks or a human decision.
Two gates carry most of the safety load. The plan sign-off gate sits between reading the codebase and modifying it: the agent produces a written plan, and a human approves it before any file changes. And promotion is human-only: the skill flatly refuses to let the agent be the one who ships to production, no matter how green everything looks.
Gates encode a truth that pure instructions can't: the risk in agentic engineering work isn't that the agent can't do the tasks — it's that it does them in an order, or at a moment, that removes a human decision that needed to exist. When I evaluated agents without the skill on a deliberately tempting prompt — "migrate the content right now" — the baseline agent cheerfully shipped a data-migration patch with no plan and no sign-off. The skilled agent, given the identical prompt, recognized the jump-ahead request, backed up to inventory, and asked for approval at the gate. Same model, same task, opposite risk profile. The description of the skill is even written to trigger on exactly those jump-ahead phrasings, because the dangerous request is the one the skill most needs to catch.
Decision two: pitfalls became tripwires, not footnotes
The playbook's pitfall list was its most valuable content, so it got the most deliberate transformation. Each pitfall entry in the skill carries three things: the symptom (what you actually observe when it bites), the rule (what the agent must do or refuse to do), and an enforcement point (which phase's exit checklist verifies it). Pitfalls aren't ambient warnings; they're indexed by phase, and opening the relevant entries is itself a required step of the phase.
A small number of pitfalls got promoted further, into inline tripwires in the main instruction file — the ones that fire mid-work, when the agent is deep in a change and not reading reference documents. The plugin-configuration example is one: the tripwire says, in effect, "if you touched plugin config, you generate a migration now, before proceeding." Another guards caching claims: any statement that "revalidation works" must be backed by a round-trip proof — edit, publish, observe the change live — because the original engagement demonstrated that caching claims are precisely where confident wrongness lives.
The evaluation results vindicated this promotion brutally. One graded scenario handed agents a repo with a plugin-config change that requires a migration — the exact 500-shaped bug from the source engagement. The skilled agent caught it every time: 3/3. The baseline agent looked at the same change and reported "no blockers": 0/3. That's not a marginal quality difference; that's the difference between an incident and a non-event, reproduced in a test harness.
Decision three: checks became code
The playbook said things like "verify the deployed routes actually respond." A human reads that and improvises. An agent needs it executable, so the skill ships two small Python scripts — and writing them taught me that skill scripts deserve production-grade paranoia.
The first script inventories a target repository read-only: detects the framework versions, maps routes, reads environment variable names (never values) to build the gap list between current state and target stack. The second runs post-deploy smoke checks: hit the routes, compare expected statuses, retry with backoff, fail loudly on network errors instead of skipping. Both were hardened the way you'd harden anything you'll trust while not watching: no silent exception swallowing, loud exits on malformed input, and regression tests for the false-positive classes I actually hit during authoring — the smoke checker once misparsed a URL containing a query string as a status-expectation pin, which is exactly the kind of bug that erodes trust in automation the first time it cries wolf.
The principle underneath: every check the skill relies on should be either executable or verifiable, never vibes. If the playbook's verification ladder had stayed prose, the agent would "verify" the way agents verify — by asserting success. Scripts convert verification from a claim into an observation.
Step three: evals, or, don't trust your own skill
Here's the step almost everyone skips, and the one that changed my mind about what I'd built. Before trusting the skill, I ran structured evaluations: a set of scenarios with fixture repositories, each run both with the skill and without it (a baseline agent given the same task cold), graded against ground truth — including checking the git state afterward to see what each agent actually did, not what it claimed.
The aggregate — 14/14 with the skill, 7/14 baseline — is less interesting than where the halves live. The baseline didn't fail randomly. It failed on exactly the scenarios the skill was built from: it skipped the human gate under a pushy prompt, and it missed the config-change-needs-migration pitfall. In other words, the skill's value concentrated precisely where the original engagement's pain concentrated. That's the signature of a skill built from bruises rather than from imagination — and it's only visible because the evals included a baseline. Without the comparison, "the agent did fine with my skill" is unfalsifiable comfort.
Two honesty notes, because evals deserve the same rigor they impose. First, one baseline run scored suspiciously well on a knowledge scenario — and inspection showed it had read the skill's own reference files, which lived in the same directory tree as the test fixtures. Contaminated baseline, documented as such; the fix is isolating fixtures from the skill under test. Second, a later round added mode-compliance scenarios — the skill supports an autonomous mode (few checkpoints, run hot) and a supervised mode (per-change approval, report-only sub-agents) — and the graded runs showed the same skill on the same task producing opposite working cadences purely from the declared mode, verified in the git history: the autonomous run committed its work and delivered patches; the supervised run left changes staged and uncommitted, waiting at the approval gate. Behavior differences you can read out of git log beat behavior differences you take on faith.
The method, extracted
Strip away Next.js and the specifics, and the playbook-to-skill arc is five moves, in order:
1. Capture during, not after. Keep engagement notes with the specificity rule: real events, real symptoms, same-day. A retrospective written a month later produces generic advice; a log written at the moment of the bruise produces enforcement material.
2. Convert ordering into gates. Find the places where the sequence is the safety — where doing step four before step two removes a human decision or an irreversible-change check — and make those boundaries the agent cannot cross on its own, rather than paragraphs it might weigh.
3. Convert pitfalls into symptom-indexed tripwires. Each one: symptom, rule, enforcement point. Promote the mid-work killers into the main instruction path; leave the rest in reference files the relevant phase is required to open.
4. Convert checks into scripts. Anything the skill calls "verified" should be a program's observation, hardened against the false positives you've actually seen — because a check that cries wolf gets deleted, and a check that silently passes is worse.
5. Prove it with baselined evals. Scenario fixtures, with-skill versus without-skill, graded against ground truth including the git state. Expect the skill's wins to cluster around its source bruises; if they don't, you've written documentation with extra steps. And audit your own harness — contaminated baselines flatter you.
The meta-lesson sitting under all five: the engagement and the skill form a loop. Real work produces the playbook; the playbook becomes the skill; the skill runs the next engagement; the next engagement's surprises become new pitfall entries. The graveyard runbook was always one-directional — knowledge flowed in and nothing flowed out. A skill with an update trigger list (mine names the ecosystem changes that should prompt revision) is a runbook with a pulse.
The takeaway
The point is not that AI agents make runbooks obsolete — it's the opposite. Agents finally give runbooks a reader that reliably shows up: one that loads the document at exactly the right moment, follows procedure without deadline-induced shortcuts, and can be tested for compliance in a way no human team ever submits to. The catch is that this reader is literal-minded, so the document has to graduate from prose to procedure: gates it cannot skip, tripwires indexed by symptom, checks that run as code, and an eval harness that proves the whole package outperforms its absence.
Your next action: pick the runbook you're most tired of re-explaining — the upgrade guide, the release checklist, the onboarding doc — and audit it with one question per line: is this a description, or is it enforceable? Every line that's merely descriptive is a line the next reader, human or agent, will skip. Rewrite three of them as a gate, a tripwire, or a script. That's the seed of a skill — and unlike the document it came from, it gets more valuable every time the work comes back around.