Skip to main content

The closest thing I have to an answer

Connecting systems isn't hard anymore, so what matters is what you put in the middle. I think that separates into knowledge, decisions, and the terrain those decisions run on, and the thing you'd actually be building is whatever keeps all three true. I've built a version of it for my own tools, and there's good evidence it could make an AI agree with me more, which I don't have a full answer to.

Carlos Ospina

By Carlos Ospina

Technical Account Manager / Drupal Advisor

· 5 min read

In the previous post I wrote about a conversation in Cartagena, and the reasoning behind decisions that nobody writes down.

Start with the obvious part. If a company runs several systems that were never designed to meet, you connect them.

That's not hard anymore, and it's barely even a skill. Most things have an API. For the ones that don't, you put a thin layer in front of them, and something like FastAPI makes that layer small enough that it isn't a project. There are libraries that will then expose those endpoints to agents directly. And when your workflow tool has no connector for what you need, n8n and Activepieces both let you write your own. Connecting systems is the part everybody can do now.

So connecting isn't the idea. The idea is what you put in the middle.

Once the reports and the occupancy and the performance numbers are all arriving in the same place, something has to decide what they mean. That's the part that lives in a person today, and it's what this series is about.

The two shapes it could take

Here's how I picture it working, in two shapes.

In the first, the thing sits inside the workflow. The systems push their information into it, agents inside it do the analysis and reach a determination, and the people who actually understand the business can go in and adjust how those agents think. Not by editing code. By changing what the agent knows and what it's supposed to weigh, then watching what comes out and correcting it again.

In the second, it doesn't run anything. It's a source. An agent living somewhere else entirely, in whatever workflow tool the company already uses, reads the current context from it before deciding. The reasoning stays in one place. The acting happens wherever it happens.

Either way, somebody needs to be able to see what's being decided and step in, because none of this works if the human drifts out of it.

Three kinds of knowledge

Now, I was describing this to a friend, and partway through I realized I'd already built a version of it. Not for a hotel. For my own tooling, so it would stop depending on documentation that goes out of date.

I've written before about why those tools exist, so I won't retell it. What I hadn't noticed until I said it out loud is that what I'd built separates into three kinds of knowledge, and the separation is why it still works when the world moves.

The clearest example is a step I make the AI take before it writes anything, which is to go look for prior art. Has somebody already solved this. Can I reuse it, extend it, or do I actually need something new.

That instruction never changes. It's true in Drupal, it's true in PHP generally, it's true in JavaScript, and it would be true in a language I've never touched.

But where you look changes completely. For PHP you're in Composer packages. For Drupal you're on drupal.org looking at modules and recipes. For JavaScript you're in npm. Same reasoning, different terrain. If you bake the terrain into the instruction, you get a tool that only works in one place.

So the instruction lives in one artifact and the terrain lives in another. I've been calling that second one a process recipe, and that's all it is, the binding between a general process and a specific environment.

Underneath both of them sit the guides. The standards, the techniques, the body of knowledge about how something is done properly. How image styles work. How to configure a view. What good CSS looks like. That knowledge is real, it's general, and it goes out of date constantly, which is exactly why it can't live inside the instruction.

And then there's the decision itself. Not how a thing works, but what to do here, given everything you know. That's the one that pulls the other pieces together, and it's the hardest to capture, because it usually isn't written down anywhere.

Put the hotel next to that and it lines up.

The guides are how buildings behave. Envelope performance, what heat and humidity do, how a facade responds to orientation. The decision is when to service something, which pulls on the guides, the reports, the season, and this specific building. And the process recipe is that a property in Cartagena and one in Bogotá don't run the same systems and don't sit in the same climate. Same reasoning, different terrain. Composer and npm again.

The mapping held. I built those layers for writing software, and for myself, and they transfer to a maintenance schedule without bending.

But the three kinds aren't the point.

It's that the system you have to build is the one that keeps them true.

That's the whole reason my tooling still works. The Drupal AI module moves from 1.3 to 1.4 to 1.5 and my instruction doesn't change, because the instruction never contained the version. The guide it reaches for gets updated, by hand or by an agent that notices it has drifted, and the next run picks up the new one. Nothing about the process had to be rewritten.

Bake the specifics into the instruction instead, and you get something that works beautifully now and is quietly wrong later, with nobody noticing, because it still runs.

That failure mode is everywhere right now. It's in every prompt that names a version. It's in every automation carrying a rule somebody decided in a meeting nobody remembers. And it would be in a hotel's maintenance agent not long after somebody set it up.

The part I don't have an answer to

Research presented at CHI this year tested that directly. Give a model a stored profile of someone, then ask it to judge a situation where that person was in the wrong, and it starts telling them they weren't. Gemini went from agreeing 30% of the time to 75%. Claude Sonnet moved 33 points, a GPT model 16. Two others didn't move at all.

It was tested on personal advice, not on engineering decisions, so I'm not claiming it transfers cleanly. But the direction should bother anyone proposing to feed an AI a rich, curated account of how a company thinks.

You could build exactly the thing I'm describing and end up with a very expensive machine for confirming what the operations director already believed.

My answer to that is partial, and it comes from running this rather than testing it. The instructions tell the AI not to agree with me, plainly, in the file it reads at the start of every session. It works, and it's fragile enough that I sometimes have to remind it. The guides are partitioned small so the context stays clean and direct, the agent fetching the piece that matters instead of everything I've ever thought. And the tooling carries adversaries, separate agents whose job is to attack the work rather than finish it.

Which of those does the most, I don't know. I've never run it the other way to find out.

So that's where I am. Connecting the systems is easy and everybody can do it. What goes in the middle is the part I'm not sure about. I think it separates into knowledge, decisions, and the terrain those decisions run on, and the thing you'd actually be building is whatever keeps all three current.

Is this valid? I don't know. It needs work, it could be built a dozen ways, and I'm putting it out to see whether it survives contact with people who have tried something similar.

If you've built something like this, or tried to and hit a wall, I'd like to hear which part broke first.

Comments

Comments are open. Add yours below.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.