Building a resilient scraping pipeline: retries, rotation and rate limits

Most scrapers start as a script that works. Turning that into a pipeline that survives target changes, network blips, and traffic spikes without daily babysitting comes down to three layers: retries, rotation, and self-imposed rate limits.

Retries: fail fast on the wrong things, slow on the right ones

Not every failure deserves a retry. A malformed URL will fail the same way every time — retrying just wastes a request and an IP. A timeout or a 5xx is often transient and worth retrying with backoff. Separate these explicitly instead of wrapping everything in one generic retry loop.

RETRYABLE = {429, 500, 502, 503, 504}

def fetch_with_retry(url, proxies, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            r = requests.get(url, proxies=proxies, timeout=15)
            if r.status_code not in RETRYABLE:
                return r
        except requests.RequestException:
            pass
        time.sleep(2 ** attempt)
    raise RuntimeError(f"Failed after {max_attempts} attempts: {url}")

Rotation: rotate on failure, not just on a timer

Rotating IPs on every request is the safest default, but for targets that need session continuity, rotate specifically when a request fails or shows a warning sign — not purely on a fixed timer. This keeps successful sessions intact while still getting you a fresh identity when one goes bad.

Rate limits: throttle yourself before the target does it for you

A self-imposed concurrency cap per target, tuned below the point where error rates start climbing, is almost always more effective than maximum throughput with a high failure rate. Log your success rate against concurrency and you'll usually find a clear knee in the curve — that's your real limit, regardless of what your infrastructure could technically handle.

Put it together

None of these three are complicated in isolation. The pipelines that run unattended for months are the ones where all three are wired together — a failed request triggers rotation, which respects the concurrency cap, which is informed by the retry outcome. It's plumbing, not cleverness, and it's worth building once properly.