Three Styles

Page numbers. ?page=2&per_page=100. Easy to reason about, easy to resume.

Offsets. ?offset=100&limit=100. The same idea counted in rows.

Cursors. ?cursor=eyJpZCI6MTAwfQ. The response hands you a token for the next batch, and you follow it until it is absent.

The difference matters when the data changes while you are crawling. Page and offset paging can skip or repeat rows as records are added and removed under you. Cursors are stable, at the cost of not being able to jump to an arbitrary page.

Always have a hard stop. A paging loop with a bug that never terminates is not just your problem: it hammers someone else's server as fast as your connection allows.

Stop on an empty batch, stop on a missing next link, and stop after a maximum number of pages regardless. The third one is the one that saves you.

◈ Graded Challenge

Write collect(pages, max_pages) walking through paged responses. pages is a list where index 0 is page 1, and each entry is an object with a results list and a has_more boolean. Starting at page 1, gather the results in order. Stop when has_more is false, when results is empty, when there are no more pages available, or when you have taken max_pages pages, whichever comes first. Return the gathered results as one flat list.

Name it exactly: collect(pages, max_pages)

3 visible + 4 hidden

loading editor…

Mark this lesson complete

Signed in, your progress follows you to every device.

Next: Crawling Without Regret →

← Back