Storing News at the Edge with D1 and R2

Building a feed aggregator at scale requires careful cost and constraint management. Synergizing SQLite at the edge with distributed object storage provides a resilient, predictable architecture.

The Problem With Naive Database Aggregation

Syndicated news feeds generate a flood of incoming items. If you insert every single RSS item as a row in a relational database, you run into two problems:

1. **Row read and write costs**: Serverless databases frequently meter operations by rows inspected or written. Running 100 feeds every ten minutes can easily trigger millions of rows per month.
2. **Unbounded storage growth**: News loses immediacy within 48 hours, but bloated tables slow queries down unless aggressive partitioning and pruning run constantly.

Splitting Metadata and Payloads

Our architecture cleanly decouples the workload:

• **Cloudflare D1 (Relational Metadata)**: Contains the registry of feed sources, column preferences, active flags, and sync watermarks (HTTP ETags and Last-Modified timestamps). D1 only records state changes.
• **Cloudflare R2 (Object Storage)**: Stores raw XML payloads and normalized JSON snapshots for each feed (`feeds/{id}/latest.json`). R2 charges zero egress fees, so delivering feed snapshots to edge workers or static builders incurs zero row penalties.

The Static Core

No database call is required for readers visiting the homepage. The latest snapshot is baked into static HTML at build time, guaranteeing sub-50ms Time to First Byte (TTFB) anywhere in the world.