The Model Is Not the System

In software engineering, we often say:

Naming is hard.

There is also a classic version of it:

There are only two hard things in Computer Science:
cache invalidation and naming things.

But why is naming hard?

At first glance, it feels like a small problem. Just pick a word and move on. In practice, it rarely works like that.

When we name something, we are not just choosing a label. We are trying to define a concept.

You might start with something like this:

function processOrder(order: Order) {
  // validate
  // apply discounts
  // calculate totals
  // persist
  // send notification
}

At first, processOrder feels reasonable.

But then you look closer.

Is it really just “processing”?
Or is it validating, calculating, persisting, and orchestrating multiple steps?

The name starts to feel too vague. Or maybe too broad.

So you try something more specific:

function finalizeOrder(order: Order) {
  // ...
}

But then again — is it really “finalizing”? What about all the steps before that?

This is where naming becomes difficult.

Not because we cannot find a word, but because the thing we are naming is not clearly defined. The boundary of the concept itself is fuzzy.

This is not just a problem in code. It is how the real world works.

Take the word “fish”. It sounds simple — something that swims in water.

But then we start asking questions.

Is a whale a fish?
Is a shark a fish?
Is a salamander a fish?

The more seriously we try to define it, the less clear it becomes.

We often think we live in a deterministic world. In reality, we live with fuzzy concepts all the time. We just get used to it.

AI makes this fuzziness much more visible.

When Uncertainty Becomes an Engineering Problem

When you start using AI in practice, one thing becomes obvious very quickly:

The same input does not always produce the same output.

Even with the same prompt, the model can respond differently. The meaning might be similar, but the structure, wording, and details can vary.

Context plays a huge role here.

If you provide rich and relevant context, the output becomes much more useful. The model can align with your situation and suggest something actionable.

But when the context is vague, the output becomes generic. Sometimes it sounds correct, but it is not actually helpful.

This is why prompt engineering became important.

But if you look at it more closely, what we are really doing is not just writing better prompts. We are doing context engineering.

We are shaping the environment in which the model operates — adding constraints, providing examples, and supplying relevant data. All of this helps guide the model toward more predictable behavior.

This is the first layer of control.

But there is a deeper shift happening.

In traditional programming, we rely on a simple assumption:

Same input, same output.

For example:

function calculateTotal(items: Item[]) {
  return items.reduce((sum, item) => sum + item.price, 0);
}

Given the same input, this function will always return the same result.

This predictability is what makes systems testable and reliable. When something breaks, we can reproduce it and fix it.

AI systems break this assumption.

The mapping between input and output is no longer strict. The result can change. The behavior can drift. Testing becomes less straightforward.

So the problem shifts.

It is no longer just:

Is the model good enough?

It becomes:

Can the system produce useful results consistently enough?

From Model Usage to System Design

If consistency becomes the problem, then a single model call is not enough.

We need a system.

A useful way to think about it is this:

The model is the reasoning engine, not the application.

It is the part that can understand, generate, and make decisions. But everything around it is still software engineering.

Before the model produces an answer, we need to decide what it should know.

Where does the information come from?
When should we retrieve it?
How do we ensure the context is relevant?

This leads to patterns like Retrieval-Augmented Generation (RAG). But RAG is not just a technique — it introduces a set of engineering challenges:

• how to index data

• how to retrieve the right information

• how to control context size

• how to filter out noise

Then comes another layer.

Many tasks are not single-step. They require multiple steps and decisions.

The system needs to determine:

• what to do first

• what comes next

• whether to call a tool

• whether more information is needed

This is where planning comes in.

Then we need to evaluate the result.

The model can generate an answer, but that does not mean the answer is correct or useful. We need mechanisms to check, refine, or retry.

And when something goes wrong, we need to debug it.

Did the model misunderstand the question?
Was the context incorrect?
Did retrieval fail?
Did the system take the wrong step?

Without visibility into these steps, we cannot improve the system.

This is why observability becomes critical.

So an AI application is not just:

input → model → output

It is closer to:

input
  → retrieval
  → planning
  → reasoning
  → tool usage
  → evaluation
  → output

The model is still essential, but it is only one part of the system.

A useful analogy is this:

The model is like a CPU.

It is powerful and necessary, but on its own, it does not give you a complete application.

The real value comes from everything around it — memory, tools, planning, evaluation, and observability.

This is where most of the engineering happens.

And this leads to an important shift:

The model provides capability. The system defines the ceiling.

Controllability, Not Determinism

At this point, the problem becomes clearer.

We are not going to remove uncertainty from AI. A probabilistic system will remain probabilistic.

But that does not mean we cannot control it.

The goal is not determinism. The goal is controllability.

Controllability means we can guide the output into a useful shape. It means we know where to adjust the system when things go wrong. It means we can run the system in production with confidence.

And to get there, we rely on system design.

For example, we can reduce ambiguity with structured outputs:

type Response = {
  summary: string;
  actionItems: string[];
  confidence: "low" | "medium" | "high";
};

We can make critical steps deterministic using tools:

const weather = await getWeather({ city: "Melbourne" });

const answer = await generateResponse({
  context: weather,
  question: userQuestion,
});

We can introduce retries, constraints, and evaluation loops to make the system more robust.

All of these techniques are doing the same thing:

They keep uncertainty within a manageable range.

This changes how we think about AI.

The model is not a magic answer generator. It is a component — a powerful one, but still just a component.

It needs to be guided, constrained, and observed.

And this is where software engineering comes back.

The next stage of AI development may not be about who has the best model. It may be about who can design the best system around it.

From this perspective, AI does not replace software engineering.

It makes it more important.

Closing Thoughts

If you’ve been working with AI and feeling that it’s powerful but unpredictable, that’s not a limitation you’re imagining — it’s part of how these systems work.

The real opportunity is not just learning how to use models, but learning how to design systems around them.

That’s where a lot of the engineering work is shifting.

If you’re interested in more practical examples like this — especially around system design, data fetching, and building reliable frontend systems — you can:

• follow my YouTube channel (I walk through these ideas with real code and diagrams)

• subscribe to the newsletter for deeper breakdowns