runc, gVisor, Kata and our own microVM runtime: the same PostgreSQL and nginx, measured

Every database on KoedoDB runs in its own microVM. People ask what that costs, so we measured it: the same PostgreSQL and the same nginx, started by four runtimes on one machine, with the same limits and the same load.

One of the four, cchv, is ours: a containerd runtime that gives each container its own Cloud Hypervisor microVM. We wrote it, and we chose its settings. Read the numbers with that in mind. The scripts are public (KoedoDB/cchv, bench/runtime-comparison) so that you can repeat them.

What was compared

The machine is a rented dedicated server: Xeon E3-1270 v6 (4 cores, 8 threads), 32 GB, two NVMe disks in RAID1. PostgreSQL 18.6 and nginx 1.29, each in a container, in a second Docker that does not touch the machine’s own.

Making it fair

The first run was not fair, and we redid it. Two things matter:

• The same CPUs for every runtime. PostgreSQL gets 3 CPUs and nginx 2, by quota, for all four.
• The size of the VM. A microVM adds its own overhead on top of the container’s limit, and we checked the real VM size from the hypervisor’s command line for each run and recorded it with the results. Kata always adds one vCPU to the limit and cannot be told otherwise, so with the same 3 CPUs its VM has 4 vCPUs, one more than cchv’s 3. We show Kata both ways:
  • Same CPUs, VM one vCPU larger (3 CPUs for the container, 4 vCPUs in the VM). The extra vCPU is outside the container’s limit and the guest kernel may use it for the network and for I/O. This favours Kata.
  • Same VM size (the container’s limit lowered to 2 CPUs, so that the VM has 3 vCPUs like cchv’s). Now the VM is the same size as cchv’s, but Kata’s container can use one CPU less than cchv’s. This is unfair to Kata, and it makes cchv look better. Neither row is neutral.

The truth for Kata is between the two rows.

Each number is the median of three runs.

PostgreSQL

pgbench, 16 clients, scale 100. tpcb writes (WAL and data files), select is read-only, commit is one client committing as fast as it can.

• cchv reaches about 80% of runc’s throughput while keeping the database inside its own VM. gVisor reaches 40% on writes and reads.
• Against Kata with the VM the same size as cchv’s (Kata adds a vCPU of its own, so its container gets one CPU less: this row is unfair to Kata, on PostgreSQL and on nginx alike) and its files on virtio-fs (Kata’s default), cchv is about 1.4 times faster. That gap is mostly the file system path: see the next section, where Kata gets a block device like cchv. Against Kata with the same CPUs (and a VM one vCPU larger), Kata is a few percent ahead on throughput (4% on writes, 3% on reads, 8% on commits), while cchv’s p99 on the write test is less than half of Kata’s (8.8 ms against 19.9 ms).
• The p99 latency of the write test is the lowest on cchv.
• Where the remaining 20% goes: reading over a Unix socket inside the container, cchv matches runc. The gap is in the network path into the VM (about 12% on reads). We looked at queue counts, core scheduling and the TAP queue discipline; none of them moved it.

Kata with the data on a block device

cchv gives the database a block device. Kata’s default is virtio-fs. To see how much of the difference is that path and how much is the VM, we ran Kata again with the data on a virtio-blk disk (O_DIRECT), under the same conditions as the rest of this post (cchv v0.4.3, medians of three runs, 50 connections for nginx), in both sizes: the same CPUs for the container (the VM one vCPU larger), and the same VM size (the container one CPU smaller). As above, neither Kata row is neutral. We checked the VM size from the hypervisor’s command line for each.

• PostgreSQL. With a block device, Kata is about 11% faster than with virtio-fs on writes (4,588 against 4,127 with the same CPUs, 3,152 against 2,825 with the same VM size). With the same CPUs, Kata is 16% ahead of cchv on writes, and reads and commits are even. With the same VM size, cchv is 26% ahead on writes and 46% on reads, and commits are even. So the block device explains only a part of the 1.4 times gap to Kata’s default; the rest is the CPU that the same-VM-size row takes from Kata. The truth for Kata is between the two rows, and it is close to cchv.
• nginx, small files. Kata on a block device is 7 to 14% faster than cchv at 1 KB. Both are well above runc (31,920), which is held to its CPU quota while the vCPUs of a microVM are outside the container’s limit, so the VM runtimes can use more host CPU than the limit says. We did not measure host CPU time in this run. cchv’s speed on small files comes from the block device, and a runtime with a block device has it too.
• 100 KB and 10 MB. Kata is at 1,742 and 14, against 21,220 and 216 for cchv. The disk path does not explain it: Kata with virtio-fs is at the same level (1,640 to 1,670 and 15). It looks like the network path into the VM. We have not found the cause.
• Memory does not change with the disk path: the fixed cost of one VM is what differs.

What each one costs in memory

Memory of the host that one idle PostgreSQL costs, and the memory of many idle nginx containers side by side:

A microVM per container is not free. For a database, cchv costs about 300 MiB more than runc, and a little more than Kata. For nginx, cchv is less than half of Kata’s.

nginx

wrk, 2 CPUs, 60 seconds, requests per second:

cchv is ahead of runc on small files and behind on 10 MB files, where copying through the VM costs. The Kata rows here use virtio-fs; with a block device Kata is faster than cchv on 1 KB files (see above). We left gVisor’s 1 KB number out of this table: it is exactly 831 requests per second in every run, and 4 times that with 4 times the connections, which looks like a fixed delay in its TCP path for small responses rather than a throughput limit. We have not found the cause.

Where cchv loses

• Starting many at once. Sixteen PostgreSQL containers were all ready after 65 seconds on cchv, against 26 on Kata, 13 on gVisor and 8 on runc. One PostgreSQL starts in about 3 seconds on cchv.
• Memory per database, as above.
• Large files through nginx.
• Small files against Kata on a block device: Kata is 7 to 14% ahead at 1 KB, and 16% ahead on PostgreSQL writes when it has the same CPUs.

What these numbers do not say

One machine, with an older CPU. One PostgreSQL version, one nginx image. Kata was run with its default settings, not tuned; a tuned Kata may do better. We built cchv, so we know its settings better than the others’. Files written without fsync survived the container’s exit in all four runtimes, and fsync itself cost the same (about 200 µs) everywhere.