Network Background
hub

System Design: Service Communication

RPC, Resilience, Ordering & Delivery

Remote Procedure Calls (RPC)

Understanding the illusion of local execution in distributed systems.

compare_arrows RPC vs. REST

Philosophy
RPC (Action Oriented)

Focuses on actions. Like calling a function locally.

POST /rpc/rebootSystem
Format
REST (Resource Oriented)

Focuses on resources (nouns). Uses HTTP verbs.

POST /systems/123/reboot-jobs

"RPC tries to make network communication look like a local function call, hiding the complexity of the network."

terminal

RPC Lifecycle Simulator

laptop_mac
Client App
dns
Server
Call
Ready to call remote function...
1. client.getUser(123); // Looks local
2. Stub marshals (serializes) args
3. Network transmission...
4. Skeleton unmarshals on server
5. Server executes implementation

Handling Transient Failures

Simulate how different strategies handle a "flaky" service.

dns

Client Service

database

Target Service

Idle

System Logs

0
Success
0
Failed

Exponential Backoff

Instead of retrying immediately, wait $2^n$ seconds (plus random jitter) to avoid overwhelming a struggling server.

Circuit Breaker

If failures > threshold, "Open" the circuit (fail fast) to let the system heal. After a timeout, "Half-Open" to test if it's back.

Jitter

Adding randomness to retry intervals prevents "Thundering Herd" problems where all clients retry at the exact same millisecond.

Out-of-Order Delivery

How distributed systems reconstruct order from chaos using buffering and sequence numbers.

Sender (Strict Order)

Queue
Packet #1
Packet #2
Packet #3
Packet #4
Packet #5
Network Delay

Receiver (Re-ordering)

Reorder Buffer
Processed

How it works

The receiver maintains a variable nextExpectedId. If a packet arrives with id > nextExpectedId, it is buffered. When the missing packet arrives, the buffer is checked to see if subsequent packets can now be processed in sequence.

Guaranteed Delivery & Idempotency

Solving the "Double Charge" problem when acknowledgments get lost.

Configuration

Scenario: Payment Processing

You are a payment service. The network is unreliable and drops ACK messages often.

Account Balance

$1000
Current User Balance

Server Transaction Log

Server started. Waiting for transactions...