Open Record Step 6: A Polite Client

Add rate limiting and retries to your crawler. Sleep between requests, read any rate limit headers your API sends and log them, and honor Retry-After when you get a 429. Implement backoff that doubles and gives up after a fixed number of attempts, retrying only what is worth retrying. Then test the failure path deliberately: point the client at a URL that returns a 404 and confirm it gives up immediately rather than retrying five times, and at one that times out and confirm it backs off properly.