Distributed System Models and Goals

A distributed system is formally a collection of autonomous computing elements — call them nodes, hosts, or processes — that do not share memory or a clock, communicate only by passing messages over a network, and coordinate to give users the impression of a single, integrated computing facility. This definition has three load-bearing clauses. First, autonomy: each node can be independently started, stopped, upgraded, or destroyed. Second, no shared state: unlike threads on one machine, nodes cannot simply read each other's memory; every piece of information must travel as a message, which takes time and can be lost. Third, the illusion of coherence: the system is judged a success only if these messy, independent parts present a unified, predictable interface to whoever uses it. Contrast this with a centralized system, where a single machine (or a tightly coupled multiprocessor sharing memory and a clock) does all the work. Centralized systems are conceptually simple — there is one truth, one clock, one point of control — but they hit hard physical ceilings: a single machine has finite CPU, memory, disk, and network bandwidth, and a single point of failure. Distributing the work removes these ceilings but reintroduces two problems that centralized systems never had to face: partial failure (some nodes work while others crash) and no global clock or shared memory (nodes must infer time and state from delayed, possibly reordered messages). These two problems are the subject of their own deeper treatment elsewhere, but they are the reason distributed systems are hard, not merely "networking with extra steps." To manage this complexity, engineers describe a system's structure using an architectural model — a blueprint for who initiates communication, who serves requests, and how responsibilities are divided — and evaluate the result against a standard checklist of design goals: transparency, openness, scalability, fault tolerance, concurrency support, and resource sharing. These goals are frequently in tension (more transparency often costs performance; more openness often costs simplicity), so a real architecture is best understood as a negotiated compromise, not a checklist to maximize blindly.

Architectural models

The three foundational patterns for organizing a distributed system are client-server, peer-to-peer (P2P), and layered/tiered architectures, with most real systems being hybrids.

Client-server is the most common model. Clients initiate requests; servers wait passively, process requests, and reply. Roles are asymmetric and usually fixed: a web browser is always a client of a web server, never the reverse. This asymmetry makes the model easy to reason about and secure (servers can be locked down, clients don't need to trust each other), but it concentrates load and risk on the server side, motivating techniques like load balancing and replication.

Peer-to-peer removes the asymmetry: every node (peer) can act as both client and server, sharing resources (files, compute, bandwidth) directly with other peers. BitTorrent and blockchain networks are canonical examples. P2P systems scale well because capacity grows as peers join — each new peer brings its own resources — but they are harder to secure and coordinate, since there is no privileged node to enforce global rules.

Layered (tiered) architectures split a system vertically by function rather than by role. A classic three-tier web application has a presentation layer (UI), a logic layer (business rules), and a data layer (database), often each running on separate machines. Layering enforces separation of concerns: the logic tier can be scaled or rewritten without touching the data tier, as long as the interface between layers is preserved. Most production systems combine these: a client-server front end (browser to API server), a layered middle (API servers to application servers to databases), and sometimes a peer-to-peer back end (e.g., a distributed cache or a gossip protocol for membership).

Design goals: transparency

Transparency is the principle that a distributed system should hide its distributed nature from users and application programmers wherever possible. It is decomposed into specific sub-goals, each hiding a different aspect of "distributed-ness":

• Access transparency — clients invoke local and remote resources with the same operations; you call `read(file)` whether the file is on your disk or a server across the ocean.
• Location transparency — resources are accessed without knowing their physical location; a URL doesn't tell you which datacenter serves it.
• Migration transparency — resources can move to a new location without changing how clients refer to them.
• Relocation transparency — resources can move while being accessed, without interrupting the client.
• Replication transparency — multiple copies of a resource exist for reliability or performance, but clients see one logical resource.
• Concurrency transparency — multiple clients share resources without noticing (or being corrupted by) each other's simultaneous access.
• Failure transparency — faults are hidden or masked so clients don't need to detect and handle every failure themselves.

The pseudocode below sketches how access and location transparency are typically implemented — a client calls a stub (a local proxy) that hides whether the target is local or remote, foreshadowing how Remote Procedure Calls work in practice. ``` // Client-side view: identical call regardless of resource location function getBalance(accountId): return account_service.getBalance(accountId) // looks local // Underneath, "account_service" is a stub: function stub.getBalance(accountId): location = nameServer.resolve(accountId) // location transparency if location == LOCAL: return localAccountTable.lookup(accountId) else: message = encode("getBalance", accountId) reply = network.sendAndWait(location, message, timeout=T) if reply == TIMEOUT: return retryOrFail(accountId) // partial failure surfaces here return decode(reply) ``` The key design move is the name server / directory service: clients never hardcode "which machine," they resolve a logical name to a current physical location at call time. This single indirection is what buys location, migration, and (with retries) some failure transparency simultaneously.

Design goals: scalability

Scalability is the ability of a system to remain effective as demand grows, along three independent axes: 1. Size scalability — adding more users or resources should not degrade performance disproportionately. A system is size-scalable if the work per request stays roughly constant as the number of clients or servers grows, which usually requires avoiding centralized bottlenecks (a single lookup table, a single lock). 2. Geographical scalability — users and resources spread across large distances, where network latency (bounded below by the speed of light) becomes the dominant cost, not bandwidth or CPU. Synchronous protocols that work fine in a datacenter fail across continents. 3. Administrative scalability — the system spans multiple independent administrative domains (companies, countries, jurisdictions) with different trust levels, policies, and possibly conflicting rules. Common scaling techniques, each trading something away:

• Partitioning (sharding): split data or load across nodes by key, so no single node holds everything. Costs: cross-partition queries become expensive, and rebalancing is nontrivial.
• Replication: keep copies of data/services on multiple nodes to spread read load and improve availability. Costs: replicas can diverge, requiring a Consistency Model to define acceptable staleness.
• Caching: store recently used results close to the requester. Costs: cache invalidation — one of the two hard problems in computer science.
• Asynchronous communication: decouple request and response so slow components don't stall fast ones. Costs: harder reasoning about ordering and completion.

Design goals: openness

Openness measures whether a system is built to well-defined, published interfaces (protocols, data formats, service contracts) so that independently written components can interoperate, and so the system can be extended by third parties without touching existing code. An open system typically specifies its interfaces using an Interface Definition Language (IDL), publishes them, and separates policy (what should happen) from mechanism (how it happens) so that policies can be swapped without rewriting the mechanism. The reward for openness is portability, vendor independence, and a healthy ecosystem (think of how HTTP's openness enabled the entire web). The cost is that generality often sacrifices the performance or simplicity a closed, purpose-built system could achieve — an open system frequently needs extra negotiation, versioning, and compatibility layers.

Distributed system design is not a single algorithm with one complexity bound, but the goals above impose measurable and provable costs and limits.

Scalability cost derivation. Consider a naive centralized directory that maps every resource name to its location, consulted on every access. If there are $N$ clients issuing requests at rate $r$ each, the directory server handles load $O(N \cdot r)$, which grows linearly and unboundedly with $N$ — this server becomes the bottleneck regardless of how many other machines you add, so the system is not size-scalable. Partitioning the directory across $k$ servers by a hash of the resource name reduces per-server load to $O(N \cdot r / k)$, restoring near-linear scalability, but introduces the distinct cost of a lookup step (contacting the right shard), typically $O(1)$ expected with consistent hashing or $O(\log k)$ with structured overlays (as in Distributed Hash Tables). This is a clean illustration of the general pattern in distributed design: removing one bottleneck (centralization) introduces a new, smaller cost (routing/lookup) — there is no free elimination of work, only redistribution of it.

Transparency has a fundamental limit. Full failure transparency — making a remote call behave exactly like a local call, indistinguishable in all cases — is provably unattainable, not merely difficult. A local procedure call cannot silently vanish mid-execution the way a network packet can (the server may have processed the request and the reply may be lost, in which case retrying causes duplicate execution unless operations are idempotent or deduplicated). Any client waiting for a reply cannot, from the outside, distinguish "the server is slow" from "the server crashed" from "the network partitioned us" within finite time — this is a foundational obstruction, not an implementation gap, and is the formal seed of the impossibility results explored under Partial Failure and later, the FLP impossibility result for consensus. So "failure transparency" in practice always means masking as many failures as possible within a stated timeout and retry policy, never eliminating the distinction entirely.

Correctness argument for the name-server pattern. The stub/name-server mechanism sketched earlier is correct for location and migration transparency under one invariant: the name server's mapping is updated atomically whenever a resource moves, before the resource stops answering at its old location, or the old location forwards remaining requests until clients refresh their cached mapping. If this invariant holds, then every client eventually reaches the current location — possibly after one stale lookup and a bounded number of retries — which is the precise sense in which "migration transparency" is achieved: not that clients never notice a move, but that they never need code to handle it explicitly. If the invariant is violated (mapping updated late, no forwarding), clients can permanently address a dead location, which is exactly the class of bug seen in real outages when a service migrates without a grace period.

Honest tradeoff summary. Every transparency and scalability gain is purchased with one of: extra latency (indirection through name servers), extra storage (replicas, caches), extra complexity (retries, deduplication, versioning), or reduced consistency guarantees (stale caches, eventually-consistent replicas). A well-designed distributed system does not maximize every goal; it makes the trade-offs explicit and matches them to the application's actual requirements.

Trace a single "get profile picture" request through a hybrid system, showing which transparency goal is exercised at each hop.

Notice step 6: this is where the system's design goal (failure transparency) meets its provable limit from the previous section. The app server cannot distinguish "db-eu-3 is slow" from "db-eu-3 crashed" with certainty — it can only apply a timeout policy (say, 50 ms) and then assume failure, retrying against the replica. If the original write had actually succeeded just before the timeout, and the retry also succeeds, the system needs the write to be idempotent (or deduplicated by a request ID) to avoid a double-write. This single example row is precisely why "hide the failure" is a design goal pursued with policy, never a guarantee achieved absolutely.

Every large-scale service you use daily is an exercise in the trade-offs described here. Domain Name System (DNS) is a canonical example of a system built for geographical and administrative scalability, using a hierarchical, replicated, partitioned namespace rather than one giant table. Content Delivery Networks (Cloudflare, Akamai, Fastly) exist entirely to trade replication and consistency for geographical scalability and lower latency. Cloud platforms (AWS, Azure, GCP) sell access, location, and failure transparency as their core product — a virtual machine or a managed database "just works" while the provider silently migrates it across physical hardware. Blockchain and peer-to-peer file-sharing networks are studies in administrative scalability: they are explicitly designed to function across mutually distrusting organizations with no central authority. Microservice architectures inside a single company are layered and client-server internally, but are increasingly built on open, standardized interfaces (REST, gRPC, GraphQL) so that teams working in different languages and on different release schedules can interoperate — openness applied at organizational rather than internet scale. Understanding these goals is the prerequisite for the next layer of the subject: how such systems survive Partial Failure and reason correctly under Concurrency, and how Remote Procedure Calls implement the access-transparency illusion described here in concrete, wire-level detail.

Ask the tutor: How does the impossibility of perfect failure transparency lead directly into the FLP impossibility result for consensus?