Why some competitor pricing pages can't be scraped, and how to tell which

Some competitor pricing pages can't be scraped because they sit behind deliberate bot-detection systems like Cloudflare or Akamai — a boundary the site operator chose, not a technical bug to route around. Diagnosing which of four outcomes actually happened (a solvable JS challenge, a hard WAF block, an Akamai block, or a false alarm) determines whether there's a legitimate fix or whether the block should just be respected.

Ripplewatch tracks competitor pricing pages automatically, and every so often one comes back "Couldn't load." The instinctive next move is to assume it's a bot-detection problem and reach for a bigger hammer: rotate IPs, spoof a browser fingerprint, solve the CAPTCHA. We don't do that. Those techniques cross the line from scraping resiliently into evading a site's explicit access controls, and that's not something we're willing to build, no matter how much easier it would make our own product.

What we do instead is diagnose the actual block before deciding whether there's a legitimate fix. Requesting a page with our own identified bot user agent (RipplewatchBot, with a link back to us, not spoofed as a browser) and reading the response headers and body tells you which of a few very different things is actually happening.

Pricing signals feed directly into our Momentum score, so a page that silently stops updating doesn't just mean stale data, it means a blind spot in whether a competitor is actually speeding up.

What are the four possible outcomes when a pricing page won't load?

• Cloudflare JS challenge: an interstitial "Just a moment..." page with a cf-ray header. This one's solvable legitimately: a headless browser that actually executes JavaScript (not a fingerprint-spoofing one) clears it the same way a real visitor's browser does.
• Cloudflare WAF hard block: an explicit "Attention Required!" page. The site operator has decided automated traffic isn't welcome, full stop. We treat that as a boundary, not an obstacle.
• Akamai hard block: "Access Denied" with an x-akamai-cache-status header. Same story as a Cloudflare WAF block: a deliberate access decision, not a technical hurdle to route around.
• False positive: the page actually returns a clean 200. One pricing page we'd flagged as blocked turned out to be a stale status from a transient failure, not a real block at all. Worth checking before assuming the worst.

Cloudflare and Akamai publish their own reasoning for these classifications — see Cloudflare's bot management documentation if you want the vendor's side of what counts as a bot.

So which pricing-page failures are actually fixable?

Roughly half the scraping failures we've diagnosed this way are fixable with better tooling, namely a real headless browser instead of a plain HTTP request. The other half are a site operator's explicit choice, and no amount of engineering effort should override that. Telling the two apart before writing any code saves you from building something that either doesn't work or shouldn't exist in the first place.