Controlling Probabilistic Outcomes by Ensuring a Deterministic System

I spend more time on X/Twitter than I’d care to admit, and over the past few weeks, one debate has kept resurfacing, whether or not to read AI-generated code.

This debate has become more prominent with releases such as Fable from Anthropic, GPT-5.6 Sol from OpenAI, and Kimi K3 from Moonshot. These models can increasingly produce code that satisfies a requirement. But across the tasks I’ve used them for, one question has mattered more than whether the code works: does it belong in the existing system?

A change can work in isolation while ignoring existing boundaries, obscuring data flow, duplicating concepts, or making the next change harder. If code generation is becoming easier, I think the engineering challenge shifts toward designing constraints and feedback loops that keep generated code aligned with the larger system.

Agents amplify both sides of engineering. They amplify delivery, but they can also amplify inconsistency. If a codebase contains a clear rule, an agent can follow it at scale. If that rule remains implicit, the agent may reproduce every plausible interpretation of it at scale.

Reducing system entropy therefore means reducing the number of valid-looking but systemically wrong choices available to the agent.

As an engineer, I think it has become more important than ever to understand how the system works, where data flows, and how it transforms along the way. With everything happening around AI-generated code, this tweet from Yacine feels more relevant than ever.

To me, this points to two central responsibilities.

• The first is developing a deeper understanding of the codebase/system.
• The second is turning that understanding into constraints the model can follow and feedback signals it can learn from.

All of this converges to the central idea of this blog:

The way to make probabilistic code generation reliable is to surround it with a deterministic engineering system.

Making the System Legible and Its Boundaries Explicit

Before we can control what an agent generates, we need to make the system understandable enough that the correct choices are visible. Most codebases contain architectural decisions that are never formally stated. They live within module boundaries, data transformations, dependency patterns, and the accumulated understanding of the engineers working on the system. While an engineer may develop this mental model over time, an agent can only infer it from the context it receives.

This is where types and interfaces become especially important. Interfaces act as contracts between dependencies, modules, and architectural boundaries. They establish what one part of the system can expect from another, without requiring either side to understand every implementation detail.

Types make those contracts more precise. They define what data enters a module, what the module is allowed to return, how failures are represented, and which assumptions downstream code can safely make. Instead of leaving an agent to decide whether absence means null, an omitted field, or an empty collection, the type forces that decision to be made explicitly.

Put simply, types describe the data flowing through a system, while interfaces define how different parts of that system are allowed to interact across a boundary.

I have especially enjoyed working with TypeScript because its type system makes these contracts visible throughout the codebase. That experience has also changed how I write Python. I now make it a rule to define data crossing important boundaries with Pydantic schemas and use dataclasses for typed data within the application. Pydantic validates what enters the system, while dataclasses help keep its internal representation explicit.

However, this does not mean creating an interface around every function or introducing abstractions for their own sake. Types and interfaces are most valuable at meaningful seams in the system: between the frontend and backend, domain logic and persistence, raw input and validated data, or the application and an external service. Wrapping simple, self-contained types in additional abstractions often adds unnecessary complexity without making the system clearer.

These boundaries reduce the amount of context an agent must infer. If the inputs, outputs, allowed dependencies, and invariants are explicit, the agent can work freely inside that boundary without accidentally redefining the surrounding system.

One of the best parts of building a system on strong foundations of types and interfaces is that they provide context not only to the agent, but also to the human developing or orchestrating the system. By exposing a module’s inputs, outputs, and permitted operations, they compress context and allow us to understand the module without reading its entire implementation.

I have come to see types and interfaces as a shared language between the human and the AI agent. This reminds me of Eric Evans’s concept of Ubiquitous Language in Domain-Driven Design, which I recently encountered while watching Matt Pocock’s video Software Fundamentals Matter More Than Ever.

This is what I mean by understanding a system better, not reading every line of code, but understanding how its parts work and how its seams interact.

Enforcing Constraints

Making a system legible gives an agent a better chance of making the correct decision. It does not, however, prevent the agent from making the wrong one. Once we understand the boundaries and rules that matter to a codebase, the next step is to make those decisions enforceable.

A lot of devs currently encode architectural decisions in context files such as AGENTS.md or CLAUDE.md. These files might state that frontend code should not import database modules, domain logic should not depend on infrastructure, or packages should only be accessed through their public interfaces.

From what I’ve seen, context files such as AGENTS.md and CLAUDE.md are valuable as an index into the codebase. They provide the agent with a starting map: where relevant code lives, how tests are run, which conventions the project follows, and where to find deeper context. However, they are a poor mechanism for enforcing structural or architectural decisions. If a dependency must never cross a particular boundary, that rule should be encoded through import restrictions, linting, types, or structural checks.

Context files should help the agent navigate and understand the system; executable constraints should prevent it from violating the system.

Once a decision is stable and its violation can be recognized mechanically, it should become executable. Making a decision executable means encoding it in a check that runs whenever the code changes. Import restrictions can enforce dependency direction, lint rules can reject unsafe patterns, and structural checks can protect module boundaries. The system then rejects violations directly instead of relying on the agent to remember every rule.

This does not make context files obsolete. They remain useful for explaining why a boundary exists, documenting evolving conventions, and guiding decisions that still require judgment. But once a decision has been made and can be checked mechanically, the executable rule should become the authority.

Before discussing how linting helps, here is a quick overview of how linters work.

A linter parses source code into an Abstract Syntax Tree (AST), a structured representation of the code rather than its raw text.

Lint rules traverse this tree to identify specific syntax, operations, imports, and relationships between different parts of the program.

Although linting is often associated with formatting and stylistic consistency, the same mechanism can enforce correctness and architectural constraints.

An agent may add an asynchronous call without awaiting it or handling its rejection. An AI reviewer might catch the mistake, but a lint rule can prevent the entire class of unhandled promises.

Similarly, frontend code may import a database module that should never cross that boundary. A restricted-import rule can reject the dependency immediately.

These examples represent two different forms of executable knowledge:

• An unhandled-promise rule protects runtime correctness.
• An import-boundary rule protects system architecture.

After working with several AI systems, I have come to understand the distinction between inferential and deterministic checks.

Inferential checks rely on context and judgment.

Deterministic checks apply a known rule and always produce the same result for the same code.

This distinction matters when we consider AI review. AI review is inferential because it is valuable when a failure mode is unfamiliar, correctness depends on broader context, or a trade-off requires judgment. Linting is deterministic because it applies a known rule to a mechanically identifiable pattern.

I recently read a post by Mario Zechner (the creator of the pi coding agent), which further supports the idea I am trying to convey.

The two approaches should form a learning loop rather than compete with each other:

AI review finds a problem → an engineer identifies the failure class → a deterministic rule prevents it from recurring

A repeated review comment is often evidence that an inferential process is performing a deterministic job. If multiple pull requests receive the same warning about an unsafe API, an internal import path, or an unhandled promise, that warning should probably become a lint rule or structural check.

This is the idea behind a bug-to-lint workflow. Instead of fixing only the line that caused the bug, we ask what allowed that entire failure class to exist. If the pattern is mechanically recognizable, we encode the answer into the development environment and CI pipeline. The next agent receives feedback immediately, at the point where the mistake is introduced.

I encountered this idea last year through one of Danial Hasan’s posts on X. Since then, I’ve copied this skill and made a custom implementation for my workflows.

Not every architectural decision belongs in linting. Rules that remain contextual, judgment-heavy, or frequently changing should stay in documentation and review until they become stable enough to formalize. A noisy rule with frequent false positives will eventually be ignored, weakening trust in the entire feedback system.

I think at the end of the day, the goal is not to create the largest possible collection of constraints. It is to convert established engineering knowledge into fast, reliable feedback.

Closing the Feedback Loop with Telemetry

After creating a legible system with enforceable constraints, the agent still needs evidence from the running system.

Back in college, I took an operating systems course that placed a heavy emphasis on logging, specifically logging in C. At the time, I did not fully appreciate it. Now I understand that sensible, concise logging is one of the most important parts of a codebase.

By evidence, I mean signals that help the agent determine what to do next. Similar to how a gradient guides an optimization process, telemetry should show how the system’s actual behaviour differs from the expected outcome and point the agent toward its next adjustment.

Telemetry provides that evidence through logs, traces, and metrics. For it to be useful, it must provide context rather than noise. A useful signal should explain what operation occurred, what outcome was expected, what actually happened, and which version of the code produced it.

One common mistake is adding information simply to make logs appear more comprehensive. This often creates noise without making failures easier to understand. Each event should capture relevant details such as the entity, outcome, error code, function, module, and trace ID in a consistent format. Concise logging does not mean removing useful details. It means preserving the context required to understand a failure while eliminating repetitive noise.

Telemetry should also connect technical failures to their business meaning. Payment capture failed for order 123 gives the agent more useful context than a generic 500 Internal Server Error. Structured and correlated events make it easier to trace a failure across services and identify where the behaviour diverged.

The way I think about telemetry can be illustrated via the following diagram:

TL;DR: Telemetry turns runtime behaviour into signals an agent can use to understand failures and decide what to do next.

Final thoughts

So yeah, thanks for reading along to this blog, again as usual I’ve tried to distill my observations into a structure that might be useful to others finding their way through this changing field. To be fair, most of the ideas discussed here came from problems I encountered over the past three to five months, so I am learning these lessons in real time as well. Models will continue to improve, but the need for legible systems, executable constraints, and meaningful feedback loops will remain, pretty exciting times.