Not that long ago, we lived in a very absolute world. The choices we made were governed by those absolutes and the side-effects of those choices stayed with us a very long time; mainframe computers, encyclopedias, car purchases, on-prem data centers...

Slowly, the world has shifted. It has become more flexible. The absolutes that governed our choices have either been relaxed or completely eliminated. The side-effects of those choices no longer stay with us for the length of time they used to; laptops, Wikipedia, car leasing, The Cloud...

Business is no longer benefiting from enforcing absolutes and extending them with longevity. It's flexibility that is now rewarded in a world that becomes increasingly distributed every day.

So why are we still clinging to mental models based on absolutes, which result in analysis, requirements, and implementations of increasingly complex systems that are as flawed as the unattainable absolutes on which they are based?

Everything I Learned about Distributed Computing I Learned From the Telephone

30 years ago, when I wanted to talk to someone who was not in my immediate geo-location, I had to use something called a "telephone"... and it looked like this.

The phone was really awesome, but had some rules that had to be followed:

• When I was on a phone call, I was "blocked" from participating in another conversation (I could try to, but one conversation would inevitably suffer).
• I was literally tied to a specific area, constrained by the length of the phone cord.
• If the person I called was not home or did not pick up, I had to wait to deliver my message to them when they were available.
• The phone could only service one conversation at a time. Sure, other people could share the receiver, or climb on another phone in the same house, but one phone could not simultaneously service two separate conversations from two different numbers.

All absolutes. Then, the world shifted.

• Answering machines allowed me to deliver my message when the recipient was not available.
• Cordless phones allowed me to move freely around my house while talking to someone.
• Commoditization of the of the tel-com industry made it "affordable" to install two phone lines, thus, enabling two separate conversations to happen with two separate recipients. It also allowed me to dial up to the "world wide web" and talk on the phone at the same time.
• Car phones allowed me to talk in my car and the number was tied to my car, not my residence.
• Mobile phones allowed me to talk anywhere, including the car, and the number was tied to me, not my car or my residence.

Fire-and-forget, queues, parallelization, decentralization. The world shifted, and things became more distributed.

Where is the shift in our software analysis and implementation practices?

Eventual Consistency, Inconsistent State and Compensating Actions

If we can no longer rely on absolutes, then how can we continue to do business and write code to enable those businesses to succeed?

We need to start allowing the systems we design and implement to be put into an inconsistent state and continue to function.

Before you panic, please read on.

How we handle that inconsistent state is through compensating actions that are aligned with where business needs to be flexible. Then we write code to capitalize on these edge cases instead of writing code to try to prevent them.

• Flight overbooked? How about a $500 travel voucher or free first class upgrade for giving up your seat?
• Hotel overbooked? How about a free upgrade to a better room at this hotel or a competing hotel?
• Product sold out? What do you want to do? Immediate refund? Wait for it? Pick an alternative product we have in stock?

All three of these examples represent real-world systems being put into, and tolerating (for a length of time), inconsistent states. It's physically not possible to load more passengers on a plane than there are seats, but there you are, showing up for the flight, and the plane is oversold. These systems are being put into an inconsistent state by choice.

How would American Airlines scale its systems and stay in business if it had to absolutely, 100%, never, ever overbook a flight? The same things apply to hotels. They wouldn't be around, because you can't have absolutes at scale where it doesn't pay to enforce them. In other words, it's cheaper for American Airlines to give out travel vouchers or seat upgrades than to try to design, implement and maintain a system that prevents overbooking any flight ever (an absolute).

However, applying a "no absolutes" approach to everything is destined to fail. One size does not fit all and flexibility needs to be focused on where the business can capitalize on it most.

Now think about concert tickets. There is no "sorry, we oversold the stadium. But great news, we'll give you a seat upgrade at the next concert!". Here, it's CRITICAL for the business to enforce absolutes around having a seat for you when you show up at the stadium with your ticket.

"Real-time" decisions and enforcing absolutes become more difficult as our businesses, code, frameworks and run-times decentralize. Yet, here I am, in 2021, staring down a 3-call-deep blocking HTTP invocation stack "designed" to completely prevent selling something that might not be in stock.

We're still chasing absolutes.

In Closing

When you open the door to inconsistent state you gain flexibility and scale, and subsequently, the business gains speed and competitive advantage. However, it's crucial to explore edge cases that will arise from inconsistent state and bring them to the business as an opportunity for flexibility, rather than a road-block to functionality.

The world is distributed. It's time to stop chasing absolutes. It's no longer a race anyone can win.