Frameworks reference
HTTP, JSON, authentication, pagination and rate limits, then finding, cleaning and joining real public data without misrepresenting it.
Every API request is an HTTP request. Knowing the handful of pieces it is made of turns most API problems from mysteries into something you can read off the response.
A method, an address, some headers, and sometimes a body.
GET /search?q=kindred&page=2 HTTP/1.1
Host: api.example.org
Accept: application/json
Authorization: Bearer abc123Five you will use, and one property that matters more than the names.
GET read something, changing nothing
POST create something
PUT replace something entirely
PATCH change part of something
DELETE remove somethingThe first digit tells you who has the problem, which is usually all you need.
2xx it worked
3xx it moved
4xx you got something wrong
5xx the server got something wrong
200 OK
201 Created
204 No content
301 Moved permanently
400 Bad request
401 Not authenticated
403 Authenticated, but not allowed
404 Not found
409 Conflict
422 Understood, but invalid
429 Too many requests
500 Server error
503 Temporarily unavailableOptions on the end of a URL. Anything a person typed must be encoded.
# wrong: a space or an ampersand breaks the URL
url = "https://api.org/search?q=" + query
# right
from urllib.parse import urlencode
url = "https://api.org/search?" + urlencode({"q": query, "page": 2})Addresses name things, and methods say what to do with them. It is a convention rather than a standard, and most APIs follow it loosely.
GET /posts list them
POST /posts create one
GET /posts/42 read one
PATCH /posts/42 change part of it
DELETE /posts/42 remove it
GET /posts/42/comments things belonging to itThe five things worth finding before writing any code, in this order.
Before automating anything, see the actual response once.
curl "https://api.example.org/search?q=kindred"
curl -i "https://api.example.org/search?q=kindred"
curl -H "Authorization: Bearer $TOKEN" \
"https://api.example.org/me"Six types, and that is genuinely all of it.
{
"title": "Kindred", // string
"year": 1979, // number
"published": true, // boolean
"sequel": null, // null
"tags": ["novel"], // array
"author": { "name": "" } // object
}Parse, then check, then use. Skipping the middle step is where most crashes come from.
import json
data = json.loads(text)
# a missing key raises; a default does not
title = data.get("title", "")
# nested access, safely
name = (data.get("author") or {}).get("name", "")
# a list that might be absent
for tag in data.get("tags") or []:
...Things real APIs do that tutorials do not mention.
Four you will meet, in rough order of how often.
# no auth at all: many public archives
curl "https://api.example.org/items"
# key in a header (preferred)
curl -H "X-API-Key: $KEY" …
# bearer token
curl -H "Authorization: Bearer $TOKEN" …
# key in the query string (avoid where possible)
curl "https://api.example.org/items?key=$KEY"The rules are short and the consequences are not.
# never
KEY = "sk_live_abc123"
# read it from the environment
import os
KEY = os.environ["API_KEY"]No API returns a million rows at once. Every one of them splits the results, and they all do it slightly differently.
Page numbers, offsets, or cursors.
?page=2&per_page=100 page numbers
?offset=100&limit=100 offset
?cursor=eyJpZCI6MTAwfQ cursor / tokenThe shape, with the stop condition made explicit.
results = []
page = 1
while True:
data = fetch(page)
batch = data.get("results", [])
if not batch:
break
results.extend(batch)
if not data.get("next"):
break
page += 1
if page > 500: # a hard stop, always
breakA count field is frequently an estimate, and may not match what you actually receive.
Most APIs tell you your budget in the response headers, on every response.
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 4
X-RateLimit-Reset: 1735689600
# on a 429
Retry-After: 30When something fails for a reason that might pass, wait longer each time.
import time
delay = 1
for attempt in range(5):
status = do_request()
if status < 400:
break
if 400 <= status < 500 and status != 429:
break # your fault; retrying will not help
time.sleep(delay)
delay *= 2 # 1, 2, 4, 8, 16You are a guest on someone else's infrastructure, usually for free.
Networks are unreliable in specific, repeatable ways.
Silence is the expensive failure mode when you are collecting data.
# quietly wrong: missing pages become missing rows
try:
data = fetch(page)
except Exception:
data = {"results": []}
# honest: record what failed and how much
try:
data = fetch(page)
except Exception as e:
failures.append({"page": page, "error": str(e)})
continuePublic data exists in far more places than people expect.
Five questions, before you write any code against it.
The people missing from a dataset are rarely missing at random, and this matters most for exactly the questions this platform exists to ask.
It is the most common format and the least specified one.
import csv
with open("data.csv", newline="", encoding="utf-8") as f:
for row in csv.DictReader(f):
print(row["name"])Every CSV value is a string until you convert it, and conversion is where the surprises are.
"1,234" thousands separator
"1.234" or a decimal comma, depending on locale
"12%" a unit inside the value
"$1,234" a currency symbol
"N/A" not a number
"" empty, meaning unknown
"2026-08-15" a date, as textMost cleaning is these six, in roughly this order.
The single most consequential cleaning decision, and the one most often made by accident.
# these are all "missing" in real files
None, "", "N/A", "NA", "-", "null", "unknown", 0, -1, 999Never clean in place. Read raw, write clean, keep both.
data/
raw/ exactly as downloaded, never edited
clean/ produced by a script
clean.py the script, in version controlCombining two datasets on a shared identifier, and the ways it goes wrong.
by_id = {row["id"]: row for row in reference}
joined = []
missing = 0
for row in main:
match = by_id.get(row["id"])
if match is None:
missing += 1
continue
joined.append({**row, **match})
print(f"joined {len(joined)}, unmatched {missing}")Grouping rows and summarising each group.
totals = {}
for row in rows:
key = row["category"]
totals[key] = totals.get(key, 0) + row["amount"]This is part of the work, not an afterthought. A number you cannot source is a number you cannot publish.
What you may actually do with what you downloaded.
Write this down while you have it, because reconstructing it later is miserable.
source: name of the publisher
url: exact address you fetched
fetched: the date you fetched it
coverage: what period and population it covers
licence: and a link to it
notes: anything odd you noticedEvery honest piece of data work says what it cannot show.
The API does not know who you are.
It knows who you are, and the answer is still no.
You are going too fast.
What came back was not JSON.
print(response.status_code)
print(response.text[:300])The file is not the encoding you assumed.
open(path, encoding="utf-8")
open(path, encoding="latin-1")
open(path, encoding="utf-8-sig") # strips a byte order markThey are strings, or they carry a symbol.
The keys do not match as literally as you think.
A reference tells you what exists. The course teaches you when to reach for it, and has you build something real while you learn.