Crawling Without Regret

Save as you go. For anything large, write each page to disk as it arrives. A failure at page four hundred should not cost you the first three hundred and ninety-nine, and it will otherwise.

Then run your analysis against the saved files rather than the API. Your analysis will be wrong several times before it is right, and re-running it should not mean re-fetching anything.

Deduplicate afterwards. Page-based paging over changing data genuinely returns the same record twice. Deduplicate by id once collecting is finished, and report how many duplicates you removed. That number tells you something real about how much the source moved while you were reading it.

Count what failed. If page 217 errored and you skipped it, that is a hole in your data. A run that reports 'collected 4,187 records; 3 pages failed' is honest. A run that reports 4,187 and nothing else is not, and the gap will not announce itself later.