Open Record Step 5: Collect It All

Write the crawler. Walk every page, save each raw response to data/raw as it arrives, and enforce a hard maximum on the number of pages. Record which pages failed and why, in a file, rather than only printing it. When it finishes, report four numbers: pages fetched, pages failed, records collected, and duplicates removed. Then compare your record count with whatever total the API claimed and write down the difference. That difference is a finding about the source, and it belongs in your writeup.