Give Your Coding Agent More Autonomy

The record you don't otherwise have

A while back I wrote a breathless post about building this pipeline, turning permissions off, and one-shotting features. I stand by the thrill. This is the sober follow-up, and it starts with a boring fact: when an agent runs with permissions off, you are not watching. That's the point of it and the whole problem.

An agent that goes quiet leaves one artifact behind, the trace. Don't read it and you're trusting that no error meant nothing went wrong.

And things can definitely go wrong folks! Here are two that my traces caught:

• A subagent spawned with an empty message replied, instead of failing, "It looks like your message came through empty." That has the shape of an answer. It isn't one. In an autonomous run it sails through to the next step as if work had happened.
• An identifier that doesn't exist, poliva83, got generated with full confidence and wired into config. My handle is pablooliva. poliva83 is nobody.

Checking a new model before you trust it

Once you have that record, it's the raw material for evals.

Claude Code's default model moves under me. Anthropic publishes what to expect from a new model, and I read those notes. But that's the average case across everyone's usage. It says nothing about mine.

My setup has special-cased skills, a stack of CLAUDE.md rules, workflows that lean on the model behaving a specific way. A model can win every published benchmark and still get quietly worse at the one odd thing I rely on, because the release notes don't know about my one odd thing. To be clear though, there is nothing odd about my setup. People say it is the best setup. No one else has a setup like it.

Evals confirm it instead of hoping. Take a handful of your own real tasks, the ones you mined from your traces, write down what a good result looks like, and re-run them when the new model lands. Score holds, then you carry on with confidence. Score drops, the failing cases point straight at which use case regressed, before it costs you on real work. The published expectations are the prior; your evals are the confirmation. For anything you depend on, you want both.

Raising the ceiling on what you let it do unattended

This is the payoff I'd most want someone else to steal.

SDD-flow running a whole development cycle unattended only works to the degree the agent stays inside the rules I've given it: the CLAUDE.md conventions, the skills firing when they should, the plugins behaving. The more of that I can trust, the more I'll hand off without watching. So the question isn't "is autonomy safe," it's "how do I keep raising the amount I can responsibly hand off," and the answer has to beat hoping the rules still hold.

The traces close that loop with the agent itself. LangSmith's free tier keeps about 14 days, so once a fortnight I point Claude at its own record and ask it to review how it behaved. Did it follow CLAUDE.md? Did the skills and plugins fire the way they're meant to? Where did it drift?

When it finds something, it proposes the change. A rule that keeps getting ignored, reworded so it lands. A skill description that mis-fires, tightened. An ambiguous convention, made explicit. The agent holds its own behavior up against the rules it's meant to follow and tells me where the two came apart.

A concrete one. For most of the past year, Claude Code allowed only one level of subagents, an agent couldn't spawn another, and I'd baked that into my agent definitions as fact: "the Agent tool is inert inside you." Then a point release quietly enabled nested subagents, and overnight my instructions asserted something about the platform that was no longer true. I caught it by luck, I happened to see a tweet about nested subagents and went digging. The fix was easy, turn a fact that can expire into a rule with a reason. The catch was the hard part, and it hung on me seeing the right tweet. The review loop is how you find that kind of drift when you don't get lucky.

This compounds. Every pass tightens the setup, so the next autonomous run starts from firmer ground, so I can hand off a little more. The traces aren't only a safety net under autonomy. They're how you earn more of it.

What this actually takes

Less than it sounds. Turn tracing on, an environment variable, or let LangChain's LangSmith CLI skills do the wiring for you: one command drops trace, dataset, and evaluator straight into Claude Code, which line up with the three payoffs above.

npx skills add langchain-ai/langsmith-skills --agent claude-code --skill '*' --yes --global

Read it on a schedule instead of letting the dashboard become a tab you never open. Pull a handful of real tasks into an eval set so the next model upgrade meets some resistance. Then let the agent review its own traces and hand you the diffs to your config.

I still run with the guardrails off. The difference is I no longer treat a quiet run as a clean one, and I've got an agent that helps me find where it wasn't. The agent self corrects, and I get to kick back and make Trump jokes... the best jokes, no one else has better jokes.