How dbferry streams backups without touching disk

The pipeline, end to end

Conceptually the whole thing is a Unix pipe:

pg_dump -Fc -Z0 | zstd | age -r <recipient> | s3 multipart-upload

Each stage's stdout is the next stage's stdin. Data flows through in chunks; no stage buffers the whole database. In Go this is io.Readers composed end to end — the dump's output pipe feeds a zstd encoder, whose output feeds the age encryption writer, whose output is sliced into parts and uploaded. The bytes only exist in flight.

Why not just dump to a file first?

The obvious alternative — pg_dump to a temp file, compress it, encrypt it, upload it — needs scratch space as large as the (compressed, then encrypted) dump, times the intermediate copies. On a backup box running many databases, that's a disk-capacity and cleanup problem, and it leaves plaintext dumps sitting on disk between steps. Streaming removes the temp files entirely: there's never a plaintext dump at rest, and disk size stops being a limit on database size.

Stage 1 — dump: pg_dump -Fc -Z0

Custom format (-Fc) instead of plain SQL, because it allows selective restore of individual tables and parallel pg_restore -j later. Built-in compression is turned off (-Z0) on purpose — compressing twice wastes CPU, and we want zstd doing that job at a level we control. For MySQL the equivalent stage is mysqldump --single-transaction.

Stage 2 — compress: zstd

zstd gives a strong ratio at high speed and streams naturally. It sits between the dump and encryption so that what gets encrypted (and paid for in storage) is already small. Compression before encryption is the right order: ciphertext is incompressible, so you must compress first.

Stage 3 — encrypt: age, client-side, BYOK

Encryption happens on the machine taking the backup, before a byte reaches object storage. dbferry uses age — X25519 key agreement with ChaCha20-Poly1305, an authenticated (AEAD) cipher. Backups are encrypted to your age recipient; the service only ever holds the public half.

Stage 4 — upload: S3 multipart, bounded memory

The encrypted stream is uploaded with S3 multipart upload, which is what makes "stream a database of unknown size" work: you don't need to know the length up front, and you can upload part by part as bytes arrive. The defaults:

• 32 MiB parts, concurrency 4 — about 128 MiB of upload buffers in flight.
• That keeps total resident memory within a 256 MiB budget, leaving headroom for the Go runtime, zstd and age.
• The 10,000-part S3 limit puts the maximum object at ~312 GiB with these defaults; hitting it raises a controlled error before completing, suggesting a larger part size — the part size is never changed mid-upload, so memory use stays predictable.

Memory use is a function of part size and concurrency, not database size. A 5 GiB database and a 200 GiB database use the same buffers.

The validity invariant: a backup is valid only with its manifest

Alongside each ciphertext object, dbferry writes a small JSON manifest — but only after the multipart upload completes successfully. That single rule makes crashes self-healing:

• A ciphertext object without a manifest is an incomplete upload. It's never reported as success, never counted by listing, never kept by retention — it's just cleaned up.
• The manifest also carries the integrity record (a SHA-256 of the ciphertext) and the parameters used, so restore tooling can verify and read the object without dbferry running at all. The object layout is a public contract, versioned by a key_schema field.

Retention deletes ciphertext before manifest, so even a crash mid-prune leaves only a dangling manifest the next pass resolves. There's no state where a half-finished object looks like a good backup.

Why one Go binary

The pipeline, the scheduler, the job workers and the web UI are one static Go binary. Streaming composition is natural with Go's io interfaces; a single binary is trivial to ship and run yourself; and the same core code powers both the open-source CLI and the hosted service — the "we can't read your data" claim is provable because you can read the code that encrypts it.