Remote Procedure Calls
Imagine calling a function `balance = getAccountBalance(acctID)`. It silently runs on a machine across the world. The programmer writes ordinary-looking code. Underneath, a message crosses a network. A remote machine executes the work, and a reply travels back. This is the promise of the Remote Procedure Call (RPC). It hides the network behind the syntax of a local function call. Distributed programming then feels like ordinary programming. RPC matters because raw sockets are painful to program against directly. Every distributed system needs some way for one machine to ask another to do work. Examples include a database driver, a microservice mesh, a filesystem client. Another example is a blockchain node gossiping with peers. Without a clean abstraction, every team reinvents serialization formats, retry logic, and error handling. Usually, they do this badly. RPC frameworks package all of that into a reusable, largely mechanical layer. Examples include Sun RPC and CORBA, gRPC, Thrift, and Cap'n Proto. The big picture is that RPC is a transparency illusion built on top of an inherently unreliable substrate. Challenges of Distributed Systems: Partial Failure and Concurrency already establishes that networks drop messages. Machines crash mid-operation, and clocks cannot be trusted. RPC does not eliminate these problems. It cannot, by any law of physics or computability. But it gives programmers a disciplined vocabulary. This vocabulary includes marshalling, stubs, and failure semantics. It names exactly which risks remain and how to manage them. This article builds the RPC abstraction from first principles. It covers how a call is packaged (marshalling). It covers how the client-side illusion is implemented (stubs). Most importantly, it covers what "the call happened" can honestly mean. This matters when messages can be lost, duplicated, or delayed. These are the at-least-once, at-most-once, and exactly-once semantics. Understanding these guarantees precisely matters greatly. It separates an engineer who can debug production RPC failures. It separates them from one who is surprised by them.
- RPC lets a program invoke a procedure on another machine using ordinary call syntax, hiding sockets and serialization.
- Client and server stubs generated from an Interface Definition Language (IDL) handle marshalling/unmarshalling transparently.
- Marshalling converts in-memory data (with pointers, structs, byte order) into a flat, portable wire format.
- Because messages can be lost or duplicated, RPC cannot guarantee true "exactly-once" execution without extra machinery.
- Real systems choose among at-least-once, at-most-once, and (with idempotency keys or dedup) effectively-exactly-once semantics.
- RPC is the ancestor of virtually all modern distributed interfaces: gRPC, Thrift, JSON-RPC, and the Java RMI/CORBA lineage.
RPC makes remote calls look local by generating stubs that marshal arguments into messages — but no amount of syntactic sugar can make network failure semantics disappear, only name and manage them.
The idea
A Remote Procedure Call lets a client program invoke a procedure that executes on a different address space — typically a different machine — while writing code that reads exactly like a local function call: `result = f(args)`. The goal, first articulated by Birrell and Nelson in 1984, is transparency: the caller should not need to know sockets exist, and ideally should not even need to know whether the callee is local or remote. Two mechanisms make this illusion possible. Marshalling (also called serialization) is the process of flattening in-memory arguments — integers, strings, structs, arrays — into a linear sequence of bytes that can be sent over a network and reconstructed identically at the other end. Marshalling must resolve real machine differences: byte order (endianness), integer width, pointer representations (which are meaningless across address spaces and must be replaced by copied data or handles), and structure padding. The inverse operation at the receiver, unmarshalling, reconstructs the arguments (or the return value) from bytes.
Stubs are the code that performs marshalling/unmarshalling and hides it from the application. The client stub looks, to the calling code, exactly like the real procedure; internally it marshals the arguments, sends a message to the server, blocks waiting for a reply, unmarshals the result, and returns it. The server stub (sometimes called a skeleton) receives the message, unmarshals the arguments, calls the actual procedure implementation locally, marshals the result, and sends it back. Stubs are usually not hand-written — they are generated automatically by a compiler from an Interface Definition Language (IDL) specification, which declares the procedure's name, parameter types, and return type in a language-neutral way (as in Protocol Buffers' `.proto` files or Thrift's `.thrift` files). This is exactly analogous to how a compiler generates boilerplate from a schema, freeing programmers from writing repetitive, error-prone networking code by hand. The subtlety — and the reason RPC deserves careful study rather than being treated as "just a function call" — is that the underlying channel obeys the constraints already established for Partial Failure and asynchronous networks: messages can be delayed indefinitely, dropped, duplicated, or reordered, and the server can crash between receiving a request and sending a reply. A local call can fail in exactly one way (an exception is thrown, synchronously, at the call site). A remote call can fail in ways with no local analog: the request never arrives, the request arrives and executes but the reply is lost, or the server crashes mid-execution and the caller cannot tell whether the side effect happened. RPC failure semantics — at-least-once, at-most-once, and exactly-once — are the vocabulary for naming which of these outcomes a particular RPC implementation promises to avoid.
How it works
The call lifecycle
A single RPC call moves through eight conceptual steps, regardless of which concrete framework (gRPC, Thrift, Sun RPC) implements it. ``` 1. Client calls client_stub.procedure(args) -- looks like a normal call 2. Client stub marshals args into a request message 3. Client stub sends the message to the server (via a runtime/transport layer) 4. Message arrives at server; server stub unmarshals the arguments 5. Server stub calls the real procedure with the unmarshalled arguments 6. Real procedure executes and returns a result 7. Server stub marshals the result into a reply message and sends it back 8. Client stub receives the reply, unmarshals the result, returns it to the caller ``` Steps 1 and 8 are the only ones visible to the application programmer; steps 2–7 are entirely mechanical and are what stub-generation and the RPC runtime automate.
Binding: finding the server
Before step 3 can happen, the client must know where to send the message — an IP address and port, or a handle from a naming/directory service. This process is called binding. Early RPC systems (Sun RPC) used a portmapper: a well-known daemon on each machine that maps a `(program number, version number)` pair to the current port a service is listening on. Modern systems use service discovery (DNS-SD, Consul, Kubernetes Services) to the same end: decouple "what I want to call" from "which physical socket answers."
Marshalling in detail
Marshalling must handle four concerns that a local call never faces: 1. Representation differences — a big-endian server and little-endian client must agree on a canonical wire byte order (or negotiate/tag it), so marshalling defines a machine-independent external data representation (e.g., XDR in Sun RPC, protobuf's wire format, or a self-describing format like JSON). 2. Complex data structures — pointers are addresses in the caller's private memory and are meaningless remotely. Marshalling must "flatten" a linked structure by serializing the pointed-to data itself (deep copy) rather than the address, and arrays/strings need explicit lengths since there is no shared address space to bound them. 3. Call-by-reference semantics — a local call can pass a pointer and let the callee mutate the caller's memory; a remote call cannot, since there is no shared memory. RPC systems typically restrict parameters to call-by-value, or simulate "in/out" parameters by marshalling the value in, and marshalling the (possibly modified) value back in the reply. 4. Version/schema evolution — because client and server binaries are compiled and deployed independently over time, the wire format needs a way to add new optional fields without breaking old clients (protobuf's numbered, optional fields are the classic solution).
Pseudocode: generated stubs
Below is language-agnostic pseudocode for what an IDL compiler generates, using a simple example service `AccountService.getBalance(acctID: int) -> int`. ``` // ---- CLIENT STUB (generated) ---- function getBalance_clientStub(acctID): requestID = generateUniqueID() // for dedup / at-most-once message = Marshal(procedure = "getBalance", requestID = requestID, args = [acctID]) reply = null attempts = 0 while reply == null and attempts < MAX_RETRIES: SendMessage(serverAddress, message) reply = WaitForReply(requestID, timeout = T) // may time out -> null attempts += 1 if reply == null: raise RemoteCallFailedException("no reply after retries") if reply.isException: raise Unmarshal(reply.exceptionPayload) return Unmarshal(reply.resultPayload)
// ---- SERVER STUB (generated), running in a request-handling loop ---- function serverDispatchLoop(): while true: request = ReceiveMessage() requestID = request.requestID if seenBefore(requestID): // duplicate suppression resend(cachedReply(requestID)) continue try: args = Unmarshal(request.args) result = AccountService_impl.getBalance(args[0]) // real procedure reply = Marshal(requestID = requestID, resultPayload = result) catch e: reply = Marshal(requestID = requestID, isException = true, exceptionPayload = Marshal(e)) cacheReply(requestID, reply) // enables retransmission SendMessage(request.sender, reply) ``` Two details in this pseudocode are load-bearing, not incidental. The `requestID` and the server-side reply cache (`seenBefore`, `cacheReply`) are the concrete mechanism by which "at-most-once" semantics are achieved: without them, a retransmitted request due to a lost ACK would execute the procedure twice. The client's retry loop with a bounded number of attempts is the mechanism by which "at-least-once" semantics are achieved — and its absence of a reply cache is exactly why at-least-once can duplicate side effects.
The failure-semantics taxonomy
Because the network can lose the request, lose the reply, or the server can crash at any of several points, a rigorous RPC system must specify precisely what happens in each case, rather than leaving it implicit.
- At-least-once: the client retries on timeout until it gets a reply (or gives up). Guarantees the procedure ran one or more times if the client eventually gets a reply, and zero or more times if it never does. Safe only for idempotent operations (where executing twice has the same observable effect as executing once, e.g., `set x = 5`).
- At-most-once: the server deduplicates retried requests using request IDs and a reply cache (as in the pseudocode above), guaranteeing the procedure's effect is applied at most one time even if the message is retransmitted many times. This is what most production RPC frameworks target for non-idempotent operations.
- Exactly-once: the procedure runs precisely once, guaranteed, with the effect exactly once and the caller reliably informed. This is impossible to achieve with only the RPC layer over an asynchronous, lossy network, because the client can never distinguish "the server crashed before executing" from "the server executed but the reply was lost" (see Correctness below). Genuine exactly-once semantics require either coordination with a transactional resource (durable idempotency keys, distributed transactions) or restricting to idempotent operations and calling it "exactly-once effect" rather than "exactly-once execution."
Complexity & correctness
Time cost. A single successful RPC costs one network round trip plus marshalling/unmarshalling work: if a message of size bytes is serialized/deserialized in time (true of essentially all practical formats, since each field is visited once), the dominant cost is network latency (propagation + queuing) rather than CPU. So a call with no retries costs . With retries under a timeout and up to attempts, worst-case latency is before the call is declared failed, and this bound is tight — a client cannot detect success or failure faster than one round trip, by the same reasoning used to establish the Partial Failure impossibility results (you cannot distinguish "slow" from "dead" without waiting).
Space cost. The server-side reply cache used for at-most-once semantics costs space where is the number of distinct in-flight or recently-completed request IDs being tracked; in practice this is bounded with an eviction policy (e.g., keep replies for a bounded time window or bounded count per client), trading a small, bounded memory cost for the duplicate-suppression guarantee.
Correctness of at-least-once. The invariant maintained by the client retry loop is: if the client stub returns a value, the procedure executed at least once and this was the value it returned on the execution whose reply reached the client. This is guaranteed by construction — the client only returns after receiving some well-formed reply matching its requestID — but the loop makes no promise of an upper bound on the number of executions, since every retransmission on timeout may cause a fresh, independent execution of the procedure on the server. This is safe exactly when the procedure is idempotent: for an idempotent , executing one or times produces the same observable state, so the ambiguity is harmless.
Correctness of at-most-once. The invariant is: the procedure's observable side effect occurs for at most one requestID, enforced by the server checking `seenBefore(requestID)` before executing and re-sending the cached reply on a duplicate rather than re-executing. This requires (a) requestIDs unique per logical call (typically `(clientID, sequence number)` or a UUID) and (b) the cache surviving at least as long as the client might retry. The tradeoff is that this guarantee is only as strong as the cache's lifetime and the assumption that the server itself does not crash and lose the cache — if the server crashes and restarts with an empty cache, a retried request after recovery can re-execute, so true at-most-once requires the cache to be as durable as the effect it is protecting (e.g., written to the same transactional log).
Why exactly-once execution is impossible over asynchronous unreliable channels. Consider the last message the server would send: the reply. Suppose the server executes the procedure, then crashes before the reply is sent, or the reply is sent and lost in the network. From the client's perspective these two scenarios are indistinguishable: it simply observes a timeout in both. If it retries, in the first case it causes a second execution (violating "exactly once"); if it does not retry, in the second case the procedure ran but the client will never know or receive the result (violating "the caller is informed of the result"). No finite protocol built purely on send/receive over a channel that can lose the last message in a sequence can resolve this ambiguity — this is essentially the [Two Generals Problem](FLP-adjacent reasoning), and it is why practitioners are careful to say a system provides exactly-once effects</strong> (via idempotency keys checked against a durable store) rather than true exactly-once execution. This honest impossibility result is exactly why Distributed Transactions and Two-Phase Commit exists as a heavier, coordinated mechanism for cases where "maybe executed twice" truly cannot be tolerated.
Tradeoff summary. At-least-once is cheap (no server-side state) but unsafe for non-idempotent operations. At-most-once is safer but requires server-side memory and a durable cache, and still cannot promise the operation definitely ran if the client gives up before any reply arrives. Neither can offer true exactly-once without extra coordination machinery layered on top — this is a fundamental limit of the model, not an engineering gap.
Worked example
Trace a `transferFunds(from=A, to=B, amount=100)` RPC — deliberately not idempotent, since calling it twice moves 200 units, not 100 — under at-least-once semantics with no server-side deduplication, to show precisely where it breaks.
| Step | Event | Client state | Server state | Network |
|---|---|---|---|---|
| 1 | Client sends request, requestID=R7 | waiting, attempt 1 | — | request R7 in flight |
| 2 | Server receives R7, executes transfer | waiting | A: −100, B: +100 (done) | reply in flight |
| 3 | Reply from server is lost (e.g., server NIC drops it) | waiting | transfer already applied | reply dropped |
| 4 | Client timeout T expires, no reply seen | retries: sends R7 again | — | request R7 (2nd copy) in flight |
| 5 | Server receives request again — no dedup, treats as new call | waiting | A: −100 again, B: +100 again | reply in flight |
| 6 | Reply arrives this time | returns success | A has lost 200 total, B gained 200 total | — |
The client observes one successful call and one logical `transferFunds(A,B,100)` invocation in its code, yet the ledger moved 200 units — a real, silent double-spend bug, and exactly the class of error that motivated the at-most-once mechanism. Now trace the same scenario with the server-side reply cache from the pseudocode (at-most-once):
| Step | Event | Server action |
|---|---|---|
| 1 | Request R7 arrives first time | `seenBefore(R7)` is false → executes transfer, caches reply, sends it |
| 2 | Reply lost in network | (server has already cached the reply keyed by R7) |
| 3 | Client times out, retransmits R7 | `seenBefore(R7)` is true → server does not re-execute; resends cached reply |
| 4 | Client receives reply, returns success | A: −100, B: +100 — moved exactly once |
The only difference between the two traces is the single `seenBefore` check and the cached reply — and it is precisely this mechanism, not any change to the network or the retry logic, that converts a double-spend bug into a correct at-most-once transfer.
Common pitfalls & traps
| Pitfall | Why it happens | Fix |
|---|---|---|
| Assuming RPC "hides" the network completely | Marshalling and stubs hide syntax, not failure modes; a remote call can partially fail in ways a local call cannot | Always design remote APIs assuming timeouts, partial execution, and duplication are possible |
| Retrying non-idempotent operations under at-least-once with no dedup | Client-side retry logic is added without server-side request tracking | Make operations idempotent (e.g., via client-supplied idempotency keys checked server-side) before enabling retries |
| Passing raw pointers/references as RPC arguments | Code ported from a single-process design assumes shared memory | Marshal by value; explicitly copy or use handles/IDs for anything that was a pointer |
| Treating "no reply received" as "operation did not happen" | Confusing "I don't know" with "no" — the reply, not the execution, was lost | Design APIs so clients can re-query state (e.g., `getTransferStatus(requestID)`) rather than assuming failure |
| Unbounded reply cache growth on the server | Deduplication cache is never evicted, growing without bound under high request volume | Bound the cache by time window or LRU size, matched to the client's maximum retry window |
| Silent schema mismatch after independent redeploys | Client and server stubs generated from different IDL versions without compatibility rules | Use IDLs with explicit versioning/optional fields (protobuf-style) and enforce backward compatibility |
Pioneers & history
The RPC abstraction crystallized the observation that most distributed programs are really just ordinary programs whose function calls happen to cross a network — and that this crossing could, and should, be made almost invisible to the programmer.
Origins · 1 minBruce Jay Nelson and how this was discovered
Nelson's 1981 PhD dissertation at Carnegie Mellon, "Remote Procedure Call," gave the first rigorous treatment of RPC as a general programming abstraction, distinguishing it clearly from ad hoc message-passing. Working alongside Andrew Birrell at Xerox PARC, he helped design and implement one of the first practical RPC systems (Cedar RPC / the Lupine system), establishing stub generation, binding, and semantics questions that every later system — Sun RPC, DCE, CORBA, gRPC — would inherit.
Where this matters
RPC is not a historical curiosity confined to distributed-systems courses — it is the load-bearing interface layer underneath most of the internet's backend infrastructure. gRPC (Google) and Apache Thrift (originally Facebook) power internal service-to-service communication at almost every large tech company, letting thousands of independently deployed microservices call each other with generated stubs instead of hand-rolled HTTP clients. NFS (Network File System), one of the oldest and most widely deployed distributed systems in existence, is built directly on Sun RPC — every `read()` and `write()` on an NFS mount is an RPC under the hood. Java RMI and .NET Remoting brought the RPC abstraction into object-oriented languages, letting method calls cross JVM or CLR process boundaries. Distributed databases and cache layers (etcd, Redis Cluster, CockroachDB) use RPC internally for consensus messages and client queries alike. Even seemingly unrelated technologies build on RPC's core ideas: MapReduce and Data-Parallel Computing dispatches map and reduce tasks to worker machines using RPC-style calls with at-least-once retry semantics guarded by idempotent, deterministic task functions — precisely the discipline this article establishes. And when an RPC's effect absolutely cannot be duplicated or lost — a bank transfer, an order placement — systems reach for the heavier coordination machinery of Distributed Transactions and Two-Phase Commit, which is best understood as "what you build when at-most-once RPC is not strong enough."
Ask the tutor: How would you redesign the `transferFunds` RPC so that it is safe to retry under at-least-once semantics without a server-side reply cache?
Is RPC the same thing as REST or HTTP APIs?
No, though they overlap in practice. RPC is an abstraction — "call a remote procedure as if it were local" — that can be implemented over many transports, including raw TCP, HTTP/1.1, or HTTP/2 (as gRPC does). REST is an architectural style built around resources and standard HTTP verbs (GET, POST, PUT, DELETE) operating on URLs, and is not inherently procedure-oriented. Many "REST APIs" in practice are really RPC-style calls dressed in RESTish URLs. The key distinguishing feature of true RPC frameworks (gRPC, Thrift) is generated client/server stubs from an IDL, giving strongly typed, language-native call syntax, which plain REST-over-JSON typically lacks unless paired with a schema tool like OpenAPI.
Why can't we just make every RPC idempotent and avoid the whole exactly-once problem?
Many operations are naturally idempotent (`setValue(x, 5)`, `deleteFile(path)`), and designing APIs to be idempotent wherever possible is genuinely excellent advice — it sidesteps the at-least-once duplication problem entirely. But not all real-world operations can be made idempotent by their nature: "transfer $100" or "charge this credit card" are inherently effectful actions where repeating them changes the outcome. The standard solution is not to make the operation itself idempotent, but to make the request idempotent by attaching a client-generated idempotency key and having the server check a durable store for that key before applying the effect — converting a non-idempotent business operation into an idempotent request-handling procedure.
What exactly does "marshalling" solve that a shared memory system doesn't need to worry about?
In a single process, passing a struct to a function just passes a pointer, and both caller and callee share the same address space and machine representation, so no translation is needed. Across a network, none of that holds: the two machines may have different endianness, different pointer sizes, and definitely do not share memory, so a pointer value is meaningless on the other side. Marshalling explicitly flattens the data — copying pointed-to values instead of addresses, tagging lengths for variable-size data, and choosing a canonical byte order — into a self-contained byte stream that any receiving machine can reconstruct without any shared context beyond the agreed wire format.
Does using TCP (a reliable transport) make RPC semantics automatically exactly-once?
No, and this is a common and costly misunderstanding. TCP guarantees reliable, in-order delivery of bytes within a single, currently-open connection — it does not survive a connection reset, a client crash-and-reconnect, or a server crash-and-restart. Even with TCP, if the connection drops after the server executed the procedure but before the reply's bytes were fully delivered, the client sees an error and may retry over a new connection, causing a duplicate execution exactly as in the UDP case. TCP eliminates a specific class of failures (in-flight corruption/reordering within a live connection) but does nothing about the fundamental exactly-once impossibility argument, which concerns end-to-end knowledge, not per-packet delivery.
How do gRPC and similar modern frameworks handle these failure semantics in practice?
Most modern frameworks default to at-most-once for unary (single request/response) calls: if the client doesn't receive a reply, it reports a failure to the application rather than silently retrying, leaving the retry decision (and thus the idempotency responsibility) to the application layer. Frameworks like gRPC additionally support explicit retry policies with configurable idempotency assumptions, deadlines, and cancellation propagation. For streaming RPCs, sequence numbers within the stream provide ordering guarantees analogous to TCP's, but the same "did my last message actually take effect" ambiguity remains at the application level whenever a stream is interrupted and reconnected.
Why does the article say RPC gives an "illusion" of a local call — what breaks the illusion in practice?
The illusion breaks down along exactly the axes where local and remote calls genuinely differ: latency (a remote call can take milliseconds to seconds instead of nanoseconds, so blocking on it naively can stall an application), partial failure (a remote call can fail in ways with no local analog, as this article details), and call-by-reference semantics (you cannot mutate the caller's memory through a remote pointer). A well-designed RPC API is written by a programmer who understands these differences explicitly — using timeouts, retries with idempotency, and value rather than reference semantics — even though the syntax at the call site looks identical to a local function call.
What is the relationship between RPC and message passing / message queues?
RPC is a special, synchronous, request-reply pattern built on top of general message passing: the client sends one message (the request) and blocks until it receives a matching reply message, and stubs make this pattern look like a function call. Asynchronous message queues (Kafka, RabbitMQ) instead decouple sender and receiver in time — the sender does not block, and the message may be processed much later — trading the "looks like a function call" transparency for durability and load-leveling. Some systems layer RPC-style semantics on top of a queue (send a request message, later receive a correlated reply message) to get RPC's programming convenience with a queue's durability guarantees.