Don't Delegate Systems Thinking to AI
A Functioning System Is Not Necessarily a Well-Understood System
Not all parts of a system carry the same risk
Large areas of modern software are relatively straightforward: CRUD applications, internal tooling, standard APIs. AI-assisted development can accelerate this work dramatically.
Some parts require deeper shared understanding
Multi-tenancy models, orchestration behind interactive environments, anything with long-term operational and architectural consequences. Getting these wrong has dire consequences.
The real challenge is collective understanding
Implementation had accelerated faster than shared systems understanding. Parts of the platform worked successfully, but few people could describe the complete operational flow.
Where This Shows Up
Much of the industry discussion currently focuses on implementation: how much code is AI-generated, whether engineers should still review code, how autonomous agents should become.
Those are important questions. What we have found internally, however, is that the more significant challenge is often maintaining shared understanding of the systems being built.
We experienced this directly while building our interactive labs platform.
The initial delivery was commercially successful. We moved quickly to support a real customer deadline, adoption happened, and the platform itself worked well for users.
At the time, that felt like evidence that the delivery approach was working.
Later, however, we started encountering different kinds of issues:
- Longer CI execution times
- Slower troubleshooting
- Uncertainty around operational boundaries
- Fragmented understanding of how components interacted
- Increased difficulty evolving parts of the platform safely
None of these problems appeared immediately after implementation. They emerged gradually as the system evolved and more engineers interacted with it.
The Underlying Issue
The underlying issue was not that AI-generated implementation was poor quality. Much of the implementation itself was entirely reasonable.
The issue was that implementation had accelerated faster than shared systems understanding.
We found ourselves in a position where parts of the platform worked successfully in production, but relatively few people could clearly describe the complete operational flow: how users moved through the system, how components interacted, where isolation boundaries existed, or which assumptions different parts of the platform depended upon.
What Changed
That experience changed how we approach system design work.
Today, we still expect engineers to use AI aggressively during implementation. In many areas, it would make little sense not to.
What changed was the importance we placed on shared intent before implementation begins.
We now spend significantly more time aligning on architecture, constraints, and operational expectations earlier in the lifecycle. Architectural decision records became more important rather than less important.
Before substantial implementation begins, we aim to establish:
- User journeys — how people actually use the system
- System boundaries — where one component's responsibility ends
- Operational assumptions — what we expect to be true in production
- Deployment constraints — how changes safely reach users
- Failure scenarios — what happens when things go wrong
- Compatibility expectations — what contracts must be maintained
AI Is Useful Here — But Not Sufficient
This does not mean AI should be excluded from architecture discussions. In practice, we have found AI extremely useful for:
- Exploring trade-offs
- Identifying edge cases
- Generating implementation options
- Refining documentation
- Challenging assumptions
But we increasingly believe that shared systems understanding must remain intentionally human-owned.
Not because humans manually write every line of implementation, but because organisations still need people who can explain: how the system works, why it works that way, where its boundaries are, and what operational risks the business is accepting.
As implementation becomes faster, those responsibilities appear to become more important rather than less important.
