Designing systems that fail gracefully

One idea has become increasingly clear while building Oblivion: no matter how much effort goes into development, failures are unavoidable.

Servers restart. Networks become unstable. External APIs become unavailable. Unexpected inputs appear.

Eventually, something goes wrong.

Early in the project, I viewed these situations as exceptional events. Today, I consider them part of the normal operating environment.

That change in perspective influenced almost every engineering decision that followed.

Failure is not the enemy

Software engineers often strive for perfection.

Applications should never crash. Requests should never fail. Every service should always be available.

While those objectives are admirable, they rarely reflect reality.

Production environments are inherently unpredictable because they depend on systems you do not control.

The internet. Cloud providers. Operating systems. Third-party APIs. Exchange infrastructure.

Any one of them can introduce unexpected behaviour.

Designing software as though failures will never occur usually creates more fragile systems.

Assume things will go wrong

One habit I've gradually adopted is asking the same question during development: "What happens if this fails?"

Not because I expect every component to fail immediately.

But because eventually, one of them will.

If an API doesn't respond, should the application stop entirely? If market data is temporarily unavailable, should existing processes continue? If a background task crashes, does everything else need to stop as well?

Thinking about failure before it happens often leads to much simpler designs.

It forces you to separate critical operations from non-critical ones and to define how the system should behave under imperfect conditions.

Predictability matters

One lesson surprised me more than any other. Users are generally tolerant of occasional failures.

What they dislike is unpredictable behaviour.

An application that clearly reports an issue and recovers safely creates confidence. An application that silently behaves differently every time creates uncertainty.

Consistency matters even when something goes wrong.

In many situations, a predictable failure is preferable to an unpredictable success.

Recovery is part of the design

Early versions of a project often focus entirely on the happy path.

Everything works. Every request succeeds. Every dependency is available.

Real systems spend a surprising amount of time outside that ideal scenario.

Recovery therefore becomes part of the architecture itself.

How quickly should the platform retry? How should it detect that a dependency has recovered? Should users even notice that something happened?

The best recovery mechanisms are often the ones users never realize exist.

Simplicity helps under pressure

Complex systems become even more complex during incidents.

When something unexpected happens, every additional dependency, configuration option or hidden interaction makes troubleshooting more difficult.

This realization reinforced an idea that appears repeatedly throughout Oblivion.

Simple systems are easier to understand. They are easier to observe. They are easier to recover.

Elegance isn't just about writing cleaner code. It's about reducing the number of things that can fail together.

Looking back

Today, I no longer measure software quality by asking whether failures occur.

I ask different questions. Does the platform detect them quickly? Does it recover safely? Does it continue behaving predictably?

Those questions matter far more in production than achieving theoretical perfection.

Because reliable software isn't software that never encounters problems. It's software that knows how to handle them when they inevitably arrive.