Do You Even Need a Database? Exploring Alternatives for Early-Stage Applications

Databases are essentially files, whether it's a single SQLite file or a directory of files managed by PostgreSQL. The real question is whether to use a database's files or your own, especially for early-stage applications. We love databases and are developing DB Pro, a database client, but the necessity of a database depends on your application's scale. Most applications are smaller than assumed, and we tested this by building HTTP servers in Go, Bun, and Rust using different storage strategies.

## Storage Strategies

We used three flat files for users, products, and orders, each in a newline-delimited JSON format. Two HTTP endpoints were created: POST to create users and GET to fetch by ID. We benchmarked the GET path, where strategies diverged.

- **Read the file every time**: This O(n) approach reads the entire file for each request, slowing down as the file grows.

- **Load into memory**: This O(1) approach loads the file into a hash map at startup, allowing fast reads but requiring enough RAM.

- **Binary search on disk**: This O(log n) method sorts data by ID, uses a fixed-width index, and performs binary searches, offering a middle ground.

## Benchmark Results

We tested with datasets of 10k, 100k, and 1M records using wrk for load testing. Results showed:

- **Linear scan**: Performance degrades linearly with file size.

- **Binary search on disk**: Fast and consistent, outperforming SQLite with minimal performance drop as data scales.

- **SQLite**: Consistent performance but with overhead for features not always needed.

- **In-memory map**: Offers the highest throughput if data fits in RAM.

## When to Use a Database

You might not need a database until your dataset doesn't fit in RAM, you need complex queries, joins, or concurrent writes. For many applications, flat files suffice, and JSONL files can be easily migrated to a database later.

The benchmark code is available for download, allowing you to explore these strategies further.