Putting 15 million web servers on one VM sounds impossible when the machine has only 10 GB of RAM and a single CPU core. Yet a specialized engineering experiment reportedly passed that target and reached approximately 16 million lightweight server instances. The result was not achieved by tuning Nginx, raising a few limits or launching millions of containers. It required replacing much of the conventional Linux networking path with a compact Rust TCP/IP stack, userspace packet I/O and a runtime designed specifically for extreme concurrency.
The experiment is valuable because it exposes where high-concurrency servers actually spend their resources. The limiting factors are often not HTTP response bytes or application code, but sockets, file descriptors, packet buffers, scheduler state and kernel accounting. It also demonstrates why DPDK, AF_XDP and smoltcp can unlock extraordinary density while introducing substantial operational trade-offs.
Why the conventional Linux TCP stack reaches its limits
A normal Linux web server follows a proven architecture. The network interface delivers packets to the kernel, the Linux TCP stack manages connections, and applications interact with sockets through system calls. This design provides mature congestion control, routing, firewall integration, observability and isolation. It is the correct choice for nearly every ordinary backend.
At millions of endpoints or connections, however, fixed per-object costs become decisive. A listening or connected socket needs kernel structures, receive and transmit queues, protocol state and accounting metadata. Incoming packets may require additional sk_buff objects. The application also needs a file descriptor for each conventional socket, plus userspace bookkeeping.
Exact Linux socket memory usage varies by kernel version, configuration, queue limits and workload, so there is no universal per-socket figure. Even a cost of several kilobytes, however, becomes tens of gigabytes at 15 million sockets. Raising ulimit -n changes a policy limit; it does not eliminate the memory behind file tables, sockets or TCP state.
Ports create another namespace problem
A single IPv4 address has only 65,535 TCP port numbers, and not all are normally available. Fifteen million distinct listening endpoints therefore cannot each receive a unique port on one address. The design must use many destination IP addresses, multiple network namespaces, a shared listener with application-level dispatch, or a custom packet demultiplexer that treats virtual IP-and-port tuples as independent endpoints.
Conventional Linux can manage multiple addresses, but configuring millions of endpoints through normal interfaces and routing tables introduces more state. A userspace stack can instead interpret destination tuples directly without creating a Linux socket or interface object for every logical server.
The memory budget behind 15 million servers
Ten gibibytes divided by 15 million instances gives an absolute ceiling of roughly 716 bytes per instance. At 16 million, the ceiling falls to about 671 bytes. Those figures are generous because the guest operating system, page tables, packet pools, executable code, runtime and TCP/IP stack also consume memory.
- 10 GiB / 15 million: approximately 716 bytes per logical instance.
- 10 GiB / 16 million: approximately 671 bytes per logical instance.
- Practical instance budget: lower after reserving memory for the VM and shared infrastructure.
This arithmetic explains the experiment’s architecture. A server could not own a thread, process, kernel socket, large buffer or general-purpose async task allocation. It had to be represented by a tiny state record, with expensive components shared across all instances.
How smoltcp and userspace networking changed the design
smoltcp is a compact TCP/IP stack written in Rust for constrained, bare-metal and userspace environments. Instead of asking the Linux TCP stack to maintain every logical server, the experiment used userspace data structures to represent network and TCP state. Packet reception, address matching, TCP processing and minimal HTTP handling could then happen inside one controlled memory domain.
The high-level architecture looked like this:
NIC or virtual NIC
|
v
DPDK poll-mode driver or AF_XDP socket
|
v
Shared packet buffers and batch processing
|
v
smoltcp TCP/IP processing
|
v
Compact server state + minimal HTTP responseRemoving kernel sockets eliminates per-socket file descriptors and much of the associated kernel allocation. It also lets the program choose compact tables, fixed-capacity pools and purpose-built lookup structures. Rust helps enforce memory safety around those structures, although it does not automatically make networking fast or memory-efficient.
The workload was deliberately narrow. Each logical server behaved more like a tiny protocol state machine than an independent general-purpose web application. Shared response data and code avoided millions of duplicate allocations.
A custom runtime for millions of server instances
Mainstream async runtimes are efficient for realistic services, but their task headers, queues, timers, wake-up objects and heap allocations are still too expensive when the per-instance target is measured in hundreds of bytes. The experiment therefore depended on lightweight scheduling rather than one operating-system thread or conventional async task per server.
The core techniques were straightforward in principle:
- Represent each server with compact state rather than a process or thread.
- Store state densely so iteration and lookup remain cache-friendly.
- Share immutable HTTP response bytes across instances.
- Process packets in batches to amortize function calls and queue operations.
- Schedule only state machines that have received packets or reached a timer deadline.
- Avoid per-connection heap allocation wherever fixed pools or indexes can work.
One CPU core still imposes a hard processing limit. Millions of mostly idle endpoints are very different from millions of endpoints receiving sustained traffic. The reported instance count primarily demonstrates addressable state density, not the ability to serve 16 million simultaneous high-throughput requests.
Kernel networking, TUN/TAP, DPDK and AF_XDP compared
Userspace TCP stacks still need a way to exchange packets with the network device. The available mechanisms differ substantially.
Conventional kernel sockets
The normal Linux TCP stack offers the best compatibility and operational tooling. Applications use familiar socket APIs while the kernel handles routing, retransmission, congestion control and device drivers. Its generalized per-socket and per-packet overhead makes it unsuitable for this experiment’s memory target, not unsuitable for production networking.
TUN/TAP
TUN and TAP expose virtual network interfaces to userspace. They are convenient for VPNs, emulators and prototypes because packets still travel through standard Linux interfaces. Copies, system calls and kernel traversal make them easier to integrate but generally less efficient for extreme packet rates and memory density.
DPDK
DPDK maps supported device queues into userspace and commonly uses poll-mode drivers, huge pages and preallocated packet pools. The application repeatedly checks queues instead of waiting for interrupts. This can deliver predictable, high packet rates and strong control over memory, but it may dedicate CPU cores to polling and take the interface away from normal kernel networking. The DPDK Programmer’s Guide documents its memory, queue and poll-mode architecture.
AF_XDP
AF_XDP is a Linux address family connected to eXpress Data Path, or XDP. An XDP program can redirect selected packets into userspace rings backed by a registered memory area called UMEM. Zero-copy operation is available when the driver and configuration support it; otherwise AF_XDP can use copy mode. The authoritative Linux AF_XDP documentation describes the fill, completion, receive and transmit rings.
| Approach | Kernel involvement | Main advantage | Main trade-off |
|---|---|---|---|
| Linux sockets | Full TCP/IP path | Maturity and compatibility | Higher per-connection overhead |
| TUN/TAP | Kernel virtual interface | Simple userspace-stack integration | Copies and system-call costs |
| DPDK | Broad kernel bypass | Maximum packet-path control | Dedicated resources and operational complexity |
| AF_XDP | XDP-assisted fast path | Performance with Linux integration | Driver-dependent zero-copy and more kernel interaction |
DPDK vs AF_XDP: why test both?
The DPDK vs AF_XDP decision is not simply a benchmark contest. DPDK offers a more complete userspace data plane and fine-grained control over queues and packet pools. That makes it attractive for appliances, telecom systems and dedicated high performance networking services where cores and interfaces can be reserved.
AF_XDP keeps Linux in the control plane and can direct only selected traffic to userspace. Existing tools, routing components and other applications can continue using the host. This often makes deployment less disruptive, although performance depends on the network driver, queue setup, NUMA placement and whether zero-copy mode is available.
DPDK: NIC queue -> userspace driver -> smoltcp -> server table AF_XDP: NIC -> XDP program -> AF_XDP rings -> smoltcp -> server table Kernel: NIC -> Linux network stack -> socket -> application task
For the proof of concept, both fast paths addressed the same central problem: avoiding a kernel socket and file descriptor for every server instance. DPDK emphasized maximum bypass, while AF_XDP showed that a Linux-native fast path could support a similar userspace architecture.
What the reported 16 million result does—and does not—mean
The experiment reportedly reached approximately 16 million instances, exceeding its 15 million target. The achievement shows that carefully compressed state, shared resources and kernel bypass can make an otherwise impossible memory budget feasible.
It does not represent 16 million independent production websites. Real hosting workloads require variable content, logs, authentication, storage, DNS, observability and failure isolation. TLS alone adds cryptographic state, certificate handling, handshake CPU cost and buffers. Full HTTP implementations must defend against malformed requests, slow clients and resource-exhaustion attacks. Production TCP stacks also need robust congestion control, retransmission behavior, path MTU handling and extensive interoperability testing.
Nor does endpoint count equal throughput. A single core cannot execute meaningful application logic for millions of busy clients at once. The result should be read as a state-density experiment under constrained behavior, not as a replacement for ordinary web hosting benchmarks.
Lessons for backend and infrastructure engineers
The most transferable lesson is to measure the correct unit. High concurrency servers frequently fail because every connection inherits a small but repeated allocation. Inspect task headers, buffers, timers, socket queues and observability metadata before optimizing business logic.
Second, shared immutable data is powerful. If millions of logical endpoints return similar content, duplicating buffers or runtime objects wastes memory. Dense indexes and explicit state machines can be more efficient than object-heavy designs.
Third, kernel bypass should be a targeted decision. DPDK or AF_XDP may be justified for packet-processing appliances, load balancers, high-rate telemetry collectors, specialized proxies and hosting infrastructure with strict latency or density requirements. Conventional Linux networking remains preferable when compatibility, security tooling, flexible application logic and maintainability matter more than the last increment of performance.
Frequently asked questions
Did the VM really run 15 million complete websites?
No. It ran millions of extremely lightweight logical server instances under a specialized workload. They did not have the unrestricted HTTP features, TLS stacks, databases, isolation and application code expected from independent production websites.
Why could Linux tuning alone not reach the same density?
Higher file-descriptor limits and adjusted socket buffers can support more connections, but they do not remove kernel socket structures, TCP state, packet metadata or userspace task overhead. A budget below roughly 700 bytes per instance requires changing the architecture.
Is AF_XDP always slower than DPDK?
No. Results depend on drivers, hardware, queue configuration, batching, workload and zero-copy support. DPDK generally offers more direct control, while AF_XDP often provides easier integration with the Linux networking environment.
When is userspace networking worth considering?
Consider it when profiling shows the kernel packet path or per-socket state is a dominant constraint, the workload is specialized, and the organization can own a complex networking stack. It is usually unnecessary for conventional APIs, websites and business applications.
What role did Rust play?
Rust enabled compact systems programming with strong memory-safety guarantees and provided access to smoltcp. The performance gains came from data layout, batching, shared state and kernel bypass rather than from the language alone.