Capstone 11 · Backend

The Open Archive API

The APIs & Datasets course had you consume real APIs and record every place they let you down: undocumented paging, counts that lied, errors returned as HTML, no way to tell an error from data, no stated rate limit. Build the one you wished you had received. Take a public dataset with a license that permits redistribution, load it properly, and serve it. The requirements are the list of things that annoyed you. Results wrapped in an object with an honest count. Cursor pagination that is stable while the data changes. Filtering and full text search that are actually usable. Errors as JSON, always the same shape, with correct status codes. Per-key rate limiting with the remaining budget in the response headers. A version in the path so you can change it later without breaking anybody. Documentation with a working curl command that you keep working. The measure of success is not that it runs. It is that somebody else can use it to build something without asking you a single question.

Concepts used

API designCursor paginationFull text searchRate limitingCachingVersioningDocumentationDeployment

Your dataset

A real public dataset with a redistribution-permitting license, tens of thousands of rows.

Milestones

1Choose the dataset, record its license and confirm redistribution is permitted.
2Load it into PostgreSQL with a schema you designed, not a dump of the CSV columns.
3Serve a list endpoint with cursor pagination that is stable while rows are inserted.
4Add filtering and full text search, with query plans to show the indexes are used.
5Add API keys and per-key rate limiting, reporting the remaining budget in headers.
6Add caching with correct headers, and measure the hit rate.
7Write the documentation: base URL, auth, limits, paging, errors, and a curl command that works.
8Give it to one person and have them build something small against it without asking you anything.

Your workspace

Build the full project here. Work through the milestones in order, in both languages.

SANKOFA SANDBOX
loading editor…
Your turn → Build it in Python first, then rebuild it in JavaScript. The logic transfers, only the syntax changes.
← All projectsNext: The Commons →