Facts in a database. Bytes on drives.

The split

What lives where.

The metadata is small, is read far more often than it is written, must list in order, and must change atomically. Those are a database's properties, so the metadata is a database: buckets, versions, uploads, policies and cluster membership are records in a TessariDB cluster, written in its transactions and listed through its indexes.

The bytes are large and immutable once written. They go to data files on the nodes' drives, never into the metadata, and they are verified on every read.

A write

A PUT, in the order the server does it.

The answer is the last thing that happens. Until the shards are durable on their write quorum and the record has committed, the client has not been told anything.

• Verify. The SigV4 signature, the payload hash and any declared checksum are checked; a request that fails any of them is refused before a byte is stored.
• Authorise. The bucket's policy and the caller's grants are evaluated for this key.
• Encode. The body is cut into 1 MiB blocks, hashed with BLAKE3 and erasure-coded into k data and m parity shards.
• Store. Each shard goes to its node over mutual TLS and is synced; the write waits for its quorum: k shards, or k + 1 when k = m.
• Commit. The object's version record commits in TessariDB, in the same transaction as any notification event it owes.
• Answer. Only now: 200, the ETag and the checksum.

A read

Any k shards that verify.

A GET resolves the key to its current version in TessariDB, then reads from any k shards whose block hashes verify. A missing or damaged shard is rebuilt from parity for that read and listed for healing, so the reader gets the object and the cluster gets the repair.

Strong read-after-write holds for listings too: a key is visible to LIST the moment its PUT has answered, because the listing is the same index the commit wrote.

Why TessariDB

The metadata store is the hard part, and it already exists.

Ordered listing with stable continuation, conditional writes, atomic multipart completion, versioning, a membership catalog, queues for background work, topics for events and an audit record that cannot be edited: each of those is a feature an object store would otherwise build on its own. TessariDB has them as statements, so the S3 server uses them instead of reimplementing them beside the data.

The server signs in as its own account, an editor of one database that can reach nothing else. The database is reachable only by the S3 nodes.