Server to Process Fair Number of Functions

How would you design a server that has to process a fair number of functions of a requests in a second?
What if you are not aware of how many requests you will be getting?
What if requests has different priorities?

2026 Update: Fair Load Distribution — Consistent Hashing and Weighted Round Robin

Distributing work fairly across servers is a fundamental system design problem. Several algorithms exist, each with different trade-offs.

import hashlib
from bisect import bisect_left, insort

# Algorithm 1: Round Robin — equal distribution, sequential
class RoundRobin:
    def __init__(self, servers: list):
        self.servers = servers
        self.index = 0

    def get_server(self, request=None) -> str:
        server = self.servers[self.index]
        self.index = (self.index + 1) % len(self.servers)
        return server

# Algorithm 2: Weighted Round Robin — proportional distribution
class WeightedRoundRobin:
    def __init__(self, servers_weights: dict):
        """servers_weights: {'server1': 3, 'server2': 1} means 75/25 split."""
        self.sequence = []
        for server, weight in servers_weights.items():
            self.sequence.extend([server] * weight)
        self.index = 0

    def get_server(self) -> str:
        server = self.sequence[self.index]
        self.index = (self.index + 1) % len(self.sequence)
        return server

# Algorithm 3: Consistent Hashing — minimal redistribution on add/remove
class ConsistentHashRing:
    def __init__(self, virtual_nodes: int = 150):
        self.ring = {}
        self.sorted_keys = []
        self.virtual_nodes = virtual_nodes

    def _hash(self, key: str) -> int:
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_server(self, server: str) -> None:
        for i in range(self.virtual_nodes):
            virtual_key = f"{server}#{i}"
            h = self._hash(virtual_key)
            self.ring[h] = server
            insort(self.sorted_keys, h)

    def remove_server(self, server: str) -> None:
        for i in range(self.virtual_nodes):
            virtual_key = f"{server}#{i}"
            h = self._hash(virtual_key)
            if h in self.ring:
                del self.ring[h]
                self.sorted_keys.remove(h)

    def get_server(self, request: str) -> str:
        """Route request to nearest server clockwise on ring."""
        if not self.ring:
            return None
        h = self._hash(request)
        idx = bisect_left(self.sorted_keys, h)
        if idx == len(self.sorted_keys):
            idx = 0
        return self.ring[self.sorted_keys[idx]]

# Test consistent hashing
ring = ConsistentHashRing(virtual_nodes=100)
servers = ['server-A', 'server-B', 'server-C']
for s in servers:
    ring.add_server(s)

# Distribute 1000 requests
from collections import Counter
distribution = Counter(ring.get_server(f"request-{i}") for i in range(1000))
print("Consistent hashing distribution:")
for server, count in sorted(distribution.items()):
    print(f"  {server}: {count} requests ({count/10:.1f}%)")

# Removing a server: minimal redistribution
ring.remove_server('server-B')
distribution2 = Counter(ring.get_server(f"request-{i}") for i in range(1000))
print("nAfter removing server-B:")
for server, count in sorted(distribution2.items()):
    print(f"  {server}: {count} requests ({count/10:.1f}%)")

Algorithm comparison:

Round robin: Simple, equal distribution, stateless. Bad if requests have different costs
Weighted: Handles heterogeneous server capacity. Still requires knowing weights upfront
Consistent hashing: When servers are added/removed, only ~1/n of requests need redistribution (vs n/n for modulo hashing). Used by: Amazon DynamoDB, Apache Cassandra, Akamai CDN, Redis Cluster
Least connections: Route to server with fewest active connections — best for variable request duration

💡Strategies for Solving This Problem

Load Balancing and Request Handling

Got this at Meta in 2024. It's about designing scalable systems that handle variable load. Tests understanding of queuing, load balancing, and priority systems.

The Problem

Design a server that processes thousands of function requests per second. Challenges:

Unknown request rate
Different priorities
Need fairness
Can't block

Approach 1: Simple Queue

Single FIFO queue. Process requests as they come. Simple but:

No priorities
High-priority requests wait behind low-priority
Doesn't scale well

Approach 2: Priority Queue

Use heap or priority queue. High-priority requests first. Better, but:

Starvation: low-priority may never execute
Not truly "fair"

Approach 3: Multiple Queues + Round Robin

Separate queue per priority level. Use weighted round-robin to process:

60% time on high priority
30% on medium
10% on low

Prevents starvation while maintaining priorities.

Approach 4: Thread Pool with Work Stealing

Multiple worker threads. Each has local queue. If idle, "steal" work from others. Good for:

Load balancing
Utilizing all cores
Handling variable load

Key Components

Rate limiting: Prevent overwhelming server

Back pressure: Reject requests when overloaded

Monitoring: Track queue sizes, latencies

Autoscaling: Add workers when load increases

At Meta

They wanted to see if I understood the tradeoffs. Started with simple queue, they asked about priorities. Then about fairness and starvation. Finally discussed scaling and monitoring.

✅Solution

Solution: Priority Queues with Fair Scheduling

class FairRequestServer {
    constructor() {
        this.highPriorityQueue = [];
        this.mediumPriorityQueue = [];
        this.lowPriorityQueue = [];

        this.workers = [];
        this.numWorkers = 4;
        this.running = false;

        // Weighted round-robin ratios
        this.weights = { high: 6, medium: 3, low: 1 };
    }

    async start() {
        this.running = true;

        // Start worker threads
        for (let i = 0; i < this.numWorkers; i++) {
            this.workers.push(this.worker(i));
        }

        await Promise.all(this.workers);
    }

    stop() {
        this.running = false;
    }

    async addRequest(request, priority = 'medium') {
        const queues = {
            high: this.highPriorityQueue,
            medium: this.mediumPriorityQueue,
            low: this.lowPriorityQueue
        };

        const queue = queues[priority];
        if (!queue) throw new Error(`Invalid priority: ${priority}`);

        // Check if queue is too long (back pressure)
        if (queue.length > 10000) {
            throw new Error('Server overloaded - request rejected');
        }

        queue.push(request);
    }

    async worker(id) {
        console.log(`Worker ${id} started`);
        let cycle = 0;

        while (this.running) {
            const request = this.getNextRequest(cycle);

            if (request) {
                try {
                    await this.processRequest(request);
                } catch (error) {
                    console.error(`Worker ${id} error:`, error);
                }
            } else {
                // No requests, sleep briefly
                await this.sleep(10);
            }

            cycle++;
        }

        console.log(`Worker ${id} stopped`);
    }

    getNextRequest(cycle) {
        // Weighted round-robin based on cycle number
        const totalWeight = this.weights.high +
                           this.weights.medium +
                           this.weights.low;

        const position = cycle % totalWeight;

        if (position < this.weights.high) {
            // Try high priority
            if (this.highPriorityQueue.length > 0) {
                return this.highPriorityQueue.shift();
            }
        } else if (position < this.weights.high + this.weights.medium) {
            // Try medium priority
            if (this.mediumPriorityQueue.length > 0) {
                return this.mediumPriorityQueue.shift();
            }
        } else {
            // Try low priority
            if (this.lowPriorityQueue.length > 0) {
                return this.lowPriorityQueue.shift();
            }
        }

        // Fallback: try any queue
        return this.highPriorityQueue.shift() ||
               this.mediumPriorityQueue.shift() ||
               this.lowPriorityQueue.shift();
    }

    async processRequest(request) {
        // Simulate processing
        const startTime = Date.now();
        await this.sleep(request.duration || 10);
        const duration = Date.now() - startTime;

        console.log(`Processed request: ${request.id} ` +
                   `(priority: ${request.priority}, ` +
                   `duration: ${duration}ms)`);
    }

    sleep(ms) {
        return new Promise(resolve => setTimeout(resolve, ms));
    }

    getStats() {
        return {
            high: this.highPriorityQueue.length,
            medium: this.mediumPriorityQueue.length,
            low: this.lowPriorityQueue.length,
            total: this.highPriorityQueue.length +
                  this.mediumPriorityQueue.length +
                  this.lowPriorityQueue.length
        };
    }
}

// Test
async function test() {
    const server = new FairRequestServer();
    server.start();

    // Add mixed priority requests
    for (let i = 0; i < 30; i++) {
        const priority = ['high', 'medium', 'low'][i % 3];
        await server.addRequest({
            id: i,
            priority: priority,
            duration: Math.random() * 50
        }, priority);
    }

    // Let it process
    await server.sleep(2000);

    console.log("Queue stats:", server.getStats());
    server.stop();
}

test();

With Rate Limiting

class RateLimitedServer extends FairRequestServer {
    constructor() {
        super();
        this.requestCounts = new Map();  // userId -> count
        this.resetInterval = 1000;  // Reset every second

        setInterval(() => this.resetCounts(), this.resetInterval);
    }

    async addRequest(request, priority = 'medium') {
        const userId = request.userId || 'anonymous';
        const limit = request.limit || 100;  // 100 requests per second

        const count = this.requestCounts.get(userId) || 0;

        if (count >= limit) {
            throw new Error(`Rate limit exceeded for user ${userId}`);
        }

        this.requestCounts.set(userId, count + 1);
        await super.addRequest(request, priority);
    }

    resetCounts() {
        this.requestCounts.clear();
    }
}

With Autoscaling

class AutoScalingServer extends FairRequestServer {
    constructor() {
        super();
        this.minWorkers = 2;
        this.maxWorkers = 16;
        this.scaleThreshold = 100;  // Queue size to trigger scaling

        setInterval(() => this.checkScale(), 5000);
    }

    checkScale() {
        const stats = this.getStats();
        const currentWorkers = this.numWorkers;

        if (stats.total > this.scaleThreshold &&
            currentWorkers < this.maxWorkers) {
            // Scale up
            this.scaleUp();
        } else if (stats.total < this.scaleThreshold / 4 &&
                   currentWorkers > this.minWorkers) {
            // Scale down
            this.scaleDown();
        }
    }

    scaleUp() {
        if (this.numWorkers >= this.maxWorkers) return;

        console.log(`Scaling up: ${this.numWorkers} -> ${this.numWorkers + 1}`);
        this.numWorkers++;
        this.workers.push(this.worker(this.numWorkers - 1));
    }

    scaleDown() {
        if (this.numWorkers <= this.minWorkers) return;

        console.log(`Scaling down: ${this.numWorkers} -> ${this.numWorkers - 1}`);
        this.numWorkers--;
        // Note: actual worker shutdown would be more complex
    }
}

Design Tradeoffs

Approach	Pros	Cons
Single queue	Simple, fair FIFO	No priorities, poor scaling
Priority queue	Important tasks first	Starvation possible
Multiple queues	Prevents starvation	More complex
Work stealing	Good load balance	Overhead of stealing

Real-World Considerations

Monitoring:

Queue lengths
Processing latencies (p50, p99)
Error rates
Worker utilization

Circuit breakers: Fail fast when system overloaded

Timeouts: Don't let requests hang forever

Retries: With exponential backoff for transient failures

Idempotency: Safe to retry requests

Follow-Up Questions

Q: How to handle different request types (CPU vs I/O)?
A: Separate worker pools optimized for each

Q: What about distributed systems?
A: Use message queue (Kafka, RabbitMQ), load balancer

Q: How to guarantee ordering?
A: Partition by key, single worker per partition