5 signs your scraper is about to get blocked (and how to fix each one)

A block is the end of a process, not a single event. Most anti-bot systems score requests and escalate — rate limiting, then CAPTCHAs, then soft blocks, then hard bans. If you know what to watch for, you can react before you lose the target entirely.

1. Response times creeping up

A sudden, sustained increase in latency from a specific target — even with a 200 status — often means you've been shunted to a slower verification path. Track p95 latency per target, not just per request.

2. CAPTCHA rate rising

If more than a few percent of requests to a target start returning CAPTCHA pages, that's the system testing whether you're automated before it commits to a block. Rotating IPs faster and adding realistic session behavior (delays, scroll events if you're using a browser) usually buys time here.

3. Response payload size shrinking

Some sites quietly serve a stripped-down or cached version of a page to suspected bots instead of an outright block. If your parser starts finding fewer elements than usual on a page that hasn't changed, check the raw payload size against a known-good baseline.

4. Session-scoped 403s

A 403 that clears on IP rotation but reappears after N requests on the same session means the target is fingerprinting the session, not just the IP. Sticky sessions on ISP or residential IPs, capped at a lower request count, tend to fix this.

5. Your success rate drops on one ASN but not others

If you're running multiple pool types and one starts underperforming while others stay healthy, that pool's IP range has likely been partially flagged. This is exactly why it's worth keeping a fallback pool configured rather than hard-coding a single proxy type per target.

# Simple success-rate tripwire
if rolling_success_rate(target, window=100) < 0.85:
    switch_pool(target, fallback='residential')
    alert(f"{target}: success rate degraded, switched pool")

None of these signs alone means you're about to get banned. Together, trending in the same direction over a day or two, they're a reliable early warning — and they're cheap to monitor for compared to rebuilding a scraper from a hard block.