A product that works for a thousand users can fall apart at a hundred thousand, go dark on the busiest day of the year, and leave its own engineers guessing in the dark about why. Those three failures have three names, scalability, reliability and observability, and a leader who can tell them apart asks far sharper questions than one who lumps them together as "the tech needs to be solid."

The quick version

  • Scalability is whether the system keeps performing as load grows. Doubling the servers rarely doubles the throughput, shared resources and coordination eat the gains, and past a point more capacity can make a system slower.
  • Reliability is whether the system does what users need, when they need it. The modern discipline turns it into a number you choose on purpose, a target, rather than a vague wish for "no downtime."
  • Observability is whether you can understand what the system is doing from the outside, including problems nobody predicted. It is the difference between "the dashboard is green but customers are angry" and actually knowing why.
  • They reinforce each other: you can't keep a system reliable as it scales if you can't see inside it. Observability is how the other two are managed in practice.

Scalability: why adding servers hits a wall

The intuitive model of scaling is a straight line: twice the hardware, twice the work done. Reality bends that line, and the most useful map of how it bends is Neil Gunther's Universal Scalability Law (USL). Gunther's model explains throughput as a tug-of-war between three forces, the work you parallelise, the contention as requests queue for shared resources (a single database, a lock, a network link), and the coherency cost of keeping all those workers consistent with each other (Gunther, "How to Quantify Scalability", perfdynamics.com).

The sting is in the third force. Contention alone gives you diminishing returns, the curve flattens, much as Amdahl's law predicts. But coherency carries an N² cost: every new worker has to stay in sync with every other, and that overhead grows faster than the capacity you added. The consequence is counter-intuitive and worth saying plainly: beyond a certain point, adding machines can make the whole system slower, not faster. There is a peak, and you can scale past it.

The practical shift is to stop treating "throw more servers at it" as a strategy and start asking where the coordination lives. The cheapest scalability win is almost always removing a shared chokepoint, a single write database, a global lock, a service every request must touch, not buying more of everything. Before approving a capacity spend, the question to ask your engineers is: "If we double the servers, what's the one shared thing they'll all still be fighting over?" That shared thing is your real ceiling.

flowchart LR
  A(["Add more servers"]) --> B(["Linear gain
(early: each one helps)"])
  B --> C(["Contention
queueing for shared resources"])
  C --> D(["Diminishing returns
(curve flattens)"])
  D --> E(["Coherency cost
keeping workers in sync, ~N²"])
  E --> F(["Peak, then decline
more servers = slower"])
The Universal Scalability Law: capacity rises, plateaus, then can fall as coordination overhead outweighs added workers. Leaders Loop

The honest limitation: the USL is a model, not a law of physics. Its value is the shape it predicts, a peak that real systems genuinely hit, not a precise forecast of where your peak sits. Treat it as a lens that tells you what to look for (contention and coherency), and measure your own system to find out where. A model that flatters you with exact numbers it can't really know is worse than one that just points you at the right enemy.

Reliability: make it a number you choose, not a wish

Most reliability conversations go nowhere because "the system should be up" is not a target anyone can manage to. The contribution that changed this is Google's Site Reliability Engineering practice, which reframes reliability as something you set deliberately and then spend. The mechanism is three linked terms. A service level indicator (SLI) is the thing you measure, say, the proportion of requests served successfully and quickly. A service level objective (SLO) is the target you pick for it, say, 99.9% of requests succeed over a quarter. And the error budget is simply what's left over: 100% minus the SLO (Google SRE Workbook, "Implementing SLOs").

That last idea is the quietly radical one. If your objective is 99.9% over a quarter, your error budget is 0.1%, a finite, spendable allowance for unreliability. As Google's original SRE book puts it, the gap between your target and 100% is "the 'budget' of how much 'unreliability' is remaining for the quarter" (Beyer et al., Site Reliability Engineering, ch. 3, "Embracing Risk"). When you have budget left, you ship boldly. When you've burnt it, you stop shipping features and fix stability, by an agreed policy, not an argument.

100% is the wrong reliability target for almost everything. The right number is the one your users won't notice falling short of, and an error budget is how you spend the difference.

So the move is to make the trade-off explicit and shared. The error budget dissolves the oldest fight in software, the developers who want to ship versus the operators who want stability, by giving both sides the same gauge. As long as the budget holds, releases continue; when it runs dry, everyone's job is reliability. The point isn't perfection; it's that you chose the level, you can see when you're spending it, and the response is decided in advance rather than in a panic.

The honest limitation: an SLO is only as good as the SLI behind it, and it's easy to measure the wrong thing. A service can hit "99.9% of requests returned a 200" while users are quietly furious because the responses were slow, stale, or wrong. The number reassures while the experience rots. Choose indicators that track what the user actually feels, and revisit them when complaints and dashboards disagree, because when they disagree, the dashboard is usually the one that's lying.

Observability: knowing why, not just that

That gap, green dashboards, angry users, is exactly what observability addresses. The word is borrowed, deliberately, from control theory, where it was introduced by the engineer Rudolf Kálmán: a system is observable if you can determine its internal state purely from its external outputs ("Observability", control theory). Carried into software by practitioners like Honeycomb's Charity Majors, it becomes a sharp question: "can you understand any internal state the system may get itself into, simply by asking questions from the outside?", and crucially, can you ask questions you didn't predict in advance, without shipping new code to answer them (Majors, "Observability is a Many-Splendored Definition", 2020).

That phrase, without predicting the question in advance, is what separates observability from traditional monitoring. Monitoring answers the questions you already knew to ask: is CPU high, is the disk full, did the known-bad thing happen. It is built for known failure modes. Observability is built for the unknown ones, the novel, never-seen problem that emerges only in production at scale, where the useful question is "what is different about the requests that are failing?" and you don't get to know that question ahead of time.

What this asks of a leader is investment in the ability to slice your live data by anything, which customer, which region, which version, which device, after the fact, rather than only watching a wall of pre-chosen charts. Practically, the leader's test is a fire drill: next time something breaks, time how long it takes the team to answer "who is affected and what do they have in common?" If the honest answer is "we'd have to add logging and redeploy to find out," you have monitoring, not observability. Closing that gap is what lets a small team run a system far bigger than they could otherwise hold in their heads, and it's how reliability is actually defended as you scale across more infrastructure.

flowchart TD
  A(["Something is wrong
users complain"]) --> B{"Did we predict
this failure?"}
  B -->|"Yes, known issue"| C(["Monitoring
a pre-built alert fires"])
  B -->|"No, novel issue"| D{"Can we ask new
questions of live data?"}
  D -->|"Yes"| E(["Observability
slice by user, region,
version, find the cause"])
  D -->|"No"| F(["Add logging,
redeploy, wait,
guess again"])
Monitoring catches the failures you foresaw; observability lets you investigate the ones you didn't. Leaders Loop

The honest limitation, and a live debate in the field, is the popular shorthand that observability is just "three pillars: metrics, logs and traces." Majors and others argue that's a vendor framing that confuses the tools (telemetry) with the capability (being able to answer arbitrary questions). You can buy all three pillars and still not be able to investigate a novel failure. So judge observability by what your team can find out under pressure, not by which tools appear on the invoice.

A worked example

Take a fictional online retailer, call it Marketplace, heading into its biggest sale of the year. (Illustrative figures throughout; this is a teaching example, not a real company.) Traffic is forecast at roughly ten times a normal day. The instinct is simple: ten times the web servers. But every request, on every server, writes to one shared database. Adding web servers just sends more requests to the same chokepoint faster, textbook contention, and exactly the "what will they all still fight over?" question the USL pushes you to ask. The fix isn't more web servers; it's relieving the shared resource (caching reads, queuing writes), which a capacity-only plan would have missed entirely.

On reliability, Marketplace sets an SLO before the sale: 99.9% of checkout requests should succeed within two seconds, measured across the quarter. That gives a defined error budget, about 0.1% of checkouts may fail or lag before the team must stop adding features and protect stability. During the sale a third-party payment provider starts slowing down. The dashboards stay reassuringly green, because the servers are healthy and most pages load fine, but checkouts are failing. Monitoring, watching the things they thought to watch, says all is well.

Observability is what saves the day. Because the team can slice live checkout data by attributes after the fact, they ask the question they never anticipated, "what do the failing checkouts have in common?", and within minutes see it: every failure routes through one payment provider. They fail over to a backup before the error budget is spent. Notice the sequence: the scalability thinking found the real bottleneck, the SLO and error budget made "when do we intervene?" a pre-decided fact rather than a debate, and observability turned an unforeseeable incident into a five-minute diagnosis instead of an evening of guesswork. None of the three alone would have been enough.

Frequently asked questions

What's the difference between scalability and performance?

Performance is how fast the system is for a given load, one user's experience right now. Scalability is what happens to that performance as load grows. A system can be fast for ten users and collapse at ten thousand: good performance, poor scalability. They're related but distinct, and a capacity plan needs to ask both "is it fast enough now?" and "what breaks when we grow?"

Is 100% reliability the goal?

Almost never. Chasing 100% is ruinously expensive and usually pointless, because users can't perceive the difference above a certain level, and the network and devices between you and them aren't 100% reliable anyway. The SRE approach is to choose a target your users won't notice you missing (an SLO), and treat the remainder as an error budget you can deliberately spend on shipping faster. The right number is "reliable enough," chosen on purpose.

Is observability just monitoring with a new name?

No, though they overlap and you need both. Monitoring answers questions you defined in advance, known failure modes, pre-built alerts and dashboards. Observability is the ability to ask new questions of your live system, including ones you never anticipated, without shipping new code to answer them. Monitoring tells you that something is wrong; observability helps you find out why when the cause is something no one predicted.

What are SLI, SLO and error budget in one line each?

An SLI is what you measure (e.g. the share of requests that succeed quickly). An SLO is the target you set for that measure (e.g. 99.9% over a quarter). The error budget is what's left, 100% minus the SLO, a finite allowance of unreliability you can spend before stability has to take priority over new features.

We're small, do we need any of this?

You need the thinking, not the heavy machinery. A two-person team won't run a formal error-budget policy, but choosing a rough reliability target, knowing where your one shared bottleneck is, and being able to answer "who's affected and what do they have in common?" when something breaks, those pay off at any size. The tooling scales down; the questions don't change.

Related in the Toolkit

These three ideas sit on top of the rest of the stack, they govern how the services and databases you run behave under real load, and how the pipeline that ships them can tell whether a release made things worse.

Where to go next