This essay is part of a series. Check out the overview post for its wider context.
When an agent fails, the first question is almost always about the model. Did it reason correctly? Was the prompt well constructed? Sometimes those are the right questions. But more often the better question is: what kind of world it was the agent inside of?
Agents act within bounded runtimes: tools, schemas, observation loops, state stores, and side effects. The system prompt, tool surface, and returned observations are all part of the world itself. Change the tools, and you change what can be done, what can go wrong, and what kinds of action can succeed at all.
This post is also available as a structured YAML Knowledge Object; a machine-readable version designed for thinking with. You can find it in my repo here.
Ontological Hardness
We tend to think of the model as the āactive ingredientā and the environment as scaffolding. But for an agent, the environment is the medium of action. Ontological hardness. is the degree to which that medium makes entities, actions, and consequences explicit, stable, and binding over time
An ontologically hard environment is one where what exists is clearly represented, what can be done is clearly specified, and what happens as a result is durable and enforceable. In a softer environment those bindings are weak, partial, or difficult to verify. The agent must infer more of the world for itself; not just how to complete the task, but what kind of situation it is in.
My own understanding of hardness grew out of systems thinking in the Autonomous Worlds community. The concept is developed at greater length in Hard Worlds For Little Guys, which reads agent environments through fifty years of interactive fiction and MUD design. This essay is the core idea from that argument, presented on its own terms.
Josh Starkās [^1] framework for analysing hardness asks three questions of any binding: what claim does it keep true (the cast), what enforces it (the source), and what would it cost to break (the degree).
Applied to agent environments, the cast is the binding between intended action and actual consequence. The source is whatever makes that binding hold: schemas, validation layers, state models, permissions, transaction boundaries. The degree is the how much ambiguity, repair, luck, or hidden inference it takes for action and consequence to come apart.
This shifts how we diagnose failure and how we build. When an agent succeeds only by inferring missing structure or compensating for ambiguity, that is not evidence of hard environment. It is evidence that capability is being spent on reconstructing bindings the environment failed to make explicit. That inference is expensive; it costs tokens, latency, and error probability, all spent on work the environment should have done.
Lenses and Failure Modes
Three lenses, each with its own characteristic failure mode.
Lexical hardness is representational: how clearly the environment presents the entities it contains and the actions available on them. Can the agent reliably identify what exists, what kind of thing it is, and what state it is in? Are objects named consistently, types visible distinct, and is relevant state exposed in a form the agent can actually use?
When lexical hardness is low, the environment may still contain everything the agent needs, but not in a form that makes things easy to distinguish. Objects blur together, important properties are omitted or fragmented, labels are inconsistent. State is visible in pieces rather than as a coherent whole. The common failure mode here is what we often call hallucination; but what gets attributed to the model, might be produced by the environment. Ambiguous naming, inconsistent syntax, or poor signalling about what constitutes a valid action ā these are failures of lexical hardness, not of reasoning.
Interface hardness concerns how clearly the environment specifies what can be done. Are available operations explicit? Are their preconditions and arguments well defined? Is it obvious which actions apply to which entities, under which circumstances, and in what order?
When interface hardness is low, an agent may identify the right object and still fail to act on it correctly. The available operations may be underspecified, overloaded, weakly typed, or dependent on hidden conventions. The system may expose actions without exposing the conditions under which they are admissible or the assumptions they carry. A capable model can often muddle through, but that is compensation; the environment is offloading structural clarity into model inference. The characteristic failure mode is semantic misfire. Think of an API where delete means archive in one scope and permanent removal in another; the verb is the same, the consequences are not.
World hardness concerns whether the effects of actions are durable and inspectable. Does a successful action produce a persistent state change? Are side effects bounded? Can the result be verified? Are permissions, budgets, rollback conditions, and stop rules enforced structurally rather than merely suggested?
When world hardness is low, an action may appear to succeed without establishing a reliable new state. Changes may fail silently, leak into the wrong scope, depend on assumptions about ordering, or authorisation that were never made explicit. The characteristic failure mode is brittle consequence: the agent did not fail to think; it attempted to act in a world where consequences were weakly bound.
These lenses are distinct, but entangled in practice. A failure that looks like poor action selection may begin with weak lexical representation. A consequence failure may be blamed on planning when the real issue is that the environment never exposed sufficient state to plan from. And an impressive benchmark result may tell you more about a model’s ability to compensate for a soft environment than about its true ability at the task.
A fourth dimension cuts across all three: Temporal Hardness.
Does the environment make ordering, sequencing, and the persistence of state changes legible and enforceable? The video game where enemies respawn when you re-enter a room is a temporally soft; the world forgets what has already happened. When temporal bindings are weak, an agent may act on stale state, repeat work the environment has undone, or fail to recognise that a window for action has closed.
The most dangerous configuration is high world hardness paired with low interface hardness. This is how an agent deletes your inbox. It is in a world it only partially understands.
What This Means For Evals
Current work in agent benchmarking may confound task difficulty with weaknesses in environment design. If the agent fails, we blame capability. If it succeeds, we credit intelligence. You want the task to be difficult, not the ontology.
A few probes can pull apart what evaluations currently compress.
- Cross-model variance: if only the strongest model succeeds, the environment may be leaning on model-side inference.
- Re-expression sensitivity: if small phrasing changes produce different outcomes, hardness is low.
- Repair load: the more the harness must reinterpret agent output, the softer the world.
- Interface replacement: keep the task fixed and swap the tool surface; if performance shifts, that is environment design, not capability.
Treating ontological hardness as a first-class design concern changes what evaluations measure. We can ask not just whether an agent succeeded, but what kind of world it had to traverse in get there.
An Architectural Implication
If hardness is a property of the environment, then the central design decision is where constraints live.
A speed limit sign addresses the driver. A speed bump addresses the road. A constraint placed in a prompt is advice: the model must remember it, interpret it, and choose to comply. A constraint promoted into the world’s structure is physics. It cannot be forgotten, because it was never a matter of memory.
The practical move is to promote. Take constraints currently expressed as prose instructions and reimplement them as typed schemas, validated state transitions, scoped permissions, budget caps, and stop conditions.
If an action should not be available, remove it or scope it. If a field is required, make it required. If a transition is invalid, make it impossible rather than merely discouraged. If a budget must not be exceeded, enforce it at the level of execution. If a result matters, make it persistent and inspectable. The question is not āhave we told the agent not to do this?ā It is āhave we built a world in which this cannot happen?ā
Not every environment should be made maximally hard. Some tasks are inherently ambiguous; ontological softness that belongs to the task is part of the task. But softness introduced by careless environment design is hidden tax. It forces capability to be spent recovering structure that could have been supplied directly.
Ontological hardness changes the diagnostic order. Before asking āhow smart is the model?ā, ask āwhat kind of world did we give it?ā
[^1]: Josh Stark, Atoms, Institutions, Blockchains (2019). Stark defines hardness as the capacity of a system to make something very likely to be true in the future, analysed through the cast (the specific future-facing claim), source (what enforces it), and degree (what it would cost to break it).
Jay Springett / @thejaymo
Strategist, producer, and cultural theorist. Working across technology, narrative, worldrunning, digital culture, artificial intelligence, and internet culture.
Host of the 301 second long podcast Permanently Moved, and interview show Experience.Computer
Leave a Reply