What Is Database Sharding? Splitting Data Across Machines
Database sharding splits a single dataset across multiple databases, each holding a subset of the data chosen by a shard key.
A replica scales reads but not writes; every write still hits one primary. Sharding scales writes by splitting the data itself: each shard is a separate database holding part of the data, selected by a shard key. It is powerful but invasive, queries, transactions, and operations all get harder, so it is usually a last resort.
How sharding splits data
A shard key, user ID, region, tenant, determines which shard a given row lives on. Each shard is an independent database. A query for one user touches one shard; a query across all users must fan out to every shard and combine results.
What it gives you
- Write scaling beyond a single primary.
- Smaller per-shard datasets that are faster to query and index.
- Failure isolation: one shard down affects only its slice of data.
- A path past the limits of one machine.
The costs it adds
Cross-shard queries and transactions are hard or impossible. Choosing a bad shard key creates hot spots. Rebalancing data when you add shards is a major operation. These costs are why sharding is avoided until simpler options are exhausted.
Testing sharded systems
Single-database tests will not catch shard-routing bugs or cross-shard query mistakes. CI should test against a multi-shard setup so that routing, fan-out, and rebalancing logic are exercised, not just assumed.
Deploys and migrations get harder
A schema migration must run across every shard, and a resharding operation is a careful, staged process. Pipelines coordinate and verify these multi-shard changes, because a partial migration leaves shards inconsistent.
Multi-shard test setups
Bringing up several shard instances per CI run is resource-heavy. Right-sized warm runners (as Latchkey offers) keep these realistic, multi-shard tests from dragging the pipeline.
Key takeaways
- Sharding splits data across independent databases by a shard key to scale writes.
- It makes cross-shard queries, transactions, and rebalancing significantly harder.
- Migrations and tests must cover all shards, not a single representative database.