Context Engineering: Scaling Judgement in Agentic Systems

If you've worked with AI agents for a while, you've probably had this experience:

You explain a tricky decision, work through the tradeoffs and get the result you wanted. Then another agent comes along and "fixes" the very thing you spent all that time getting right.

Now you're explaining it again.

As you add more agents, keeping everyone on the same page can become a job of its own. The agents have the code, but you're still the person who remembers why it works that way.

This is the part of context engineering I've been working on: giving agents enough of the reasoning behind a project to make better decisions about it.

The goal is to scale your judgement without having to repeat yourself all day.

Watch the video

Here's the full video on YouTube if you'd rather watch.

The Problem: Ghost Context

I call the reasoning that gets left behind ghost context.

In the film, I talk about three ways it shows up:

• The amnesiac: A new agent forgets why you made a decision and repeats an old mistake.
• The imposter: An agent makes an assumption. A few summaries later, that assumption becomes something you supposedly approved.
• The reinventor: An agent fixes one part of a workflow, but another path keeps making the same mistake.

A bigger context window doesn't help much if the reasoning was never saved, the decision is attributed to the wrong person, or the rule lives in three different places.

Here are the changes that help address those problems.

1. Make It Easy to Find the Why

I use Agent History to get back to the conversations behind the work. Link a ticket or commit to the discussion that explains it, and the next agent has somewhere useful to start.

It should be able to find the original wording, read what came before and after it, and check whether you changed your mind later.

Owner Memory has a different job. It helps surface the decisions and preferences that might matter to the task at hand, without making an agent read every conversation you've ever had.

The important part is keeping what you actually said separate from what an agent thinks you meant. Keep the source, date and conditions with the decision. Make it clear whether you've reviewed it.

An imported note isn't automatically your decision. Neither is an interpretation that several agents found helpful.

You don't need a complicated system to start. Use the tickets, documents and conversation exports you're already allowed to keep. Be deliberate about what you save, who can see it and when it gets deleted.

And remember: an old conversation tells you what was said then. It doesn't prove what's running now or give an agent permission to repeat an old action.

2. Put the Complexity Where You Can Find It

This is the same maintainability problem I wrote about when building Elder.js. Getting a project started is one thing. Keeping it understandable as it grows is another.

I call the approach colocation of complexity.

For a complicated behavior, keep the implementation, the explanation of what good looks like, the important decisions and the tests somewhere an agent can find together.

For example, changing an import rule in a prompt won't fix a validator or retry path that still uses the old rule. You need to find every path that makes that decision.

That doesn't mean putting everything in one giant file. It means the next agent shouldn't have to rediscover the whole project just to make one change correctly.

3. Don't Let an Agent Put Words in Your Mouth

Suppose you tell an agent:

"For the staging demo, accept rows without a phone number."

That's a decision about a demo. It isn't permission to remove the production validation.

Keep that condition with the decision, and test both cases. The demo should accept the row. Production should still reject it.

Agents need room to make ordinary engineering decisions. But when an agent makes its own call, it should say so instead of presenting it as yours.

When it really needs your judgement, have it bring you the context, the tradeoff and a recommendation. Then save your answer where the next agent can find it.

Start With the Question You Keep Answering

Pick one rule you keep explaining or one bug that keeps coming back.

Can the next agent find out why that rule exists? Can it tell what you decided from what another agent assumed? Can it find all the places that need to change?

If not, start there.

The companion guide goes into the implementation details for Agent History, Owner Memory and colocation of complexity. You can give it to your AI along with this prompt:

Download the PDF Markdown for your AI

The guide was compiled with AI. It's general technical information, not legal, security or compliance advice, and it doesn't guarantee a safe or correct system.

Always on your team,