Price monitoring on sites behind Akamai and Cloudflare

Competitive pricing data is public, but the large retailers that matter most put a bot manager in front of it. A solver clears the challenge once, and your own HTTP client collects the pages.

What gets in the way

A challenge on the first request

Large retail sites often sit behind Akamai Bot Manager or Cloudflare. A plain HTTP client gets a challenge script or a block page instead of the product page.

Volume

A catalogue is thousands of pages, refreshed daily or hourly. Rendering each one in a headless browser multiplies server and bandwidth cost by the size of the catalogue.

Cookies that stop working

Clearance cookies are tied to an IP and a user agent. A rotating proxy pool or an inconsistent client throws away a session that was already cleared.

Which solver applies

Pick by what the site sends back. Send the wrong type and the solver refuses instead of guessing, at no charge.

Akamai SBSD solver

  • When the product page comes back with an sbsd challenge script.
from €1.00€2.50 before level discounts

Akamai edge solver

  • When the site answers “Unusual Traffic Detected” before anything else.
from €1.00€2.50 before level discounts

Cloudflare JSD solver

  • When requests without cf_clearance are blocked.
from €1.00€2.50 before level discounts

How a job runs

  1. Request the first page of a run with a Chrome-like HTTP client on a sticky residential session.
  2. If it is challenged, solve once and keep the cookies and the user agent that come back.
  3. Fetch the rest of the catalogue on that same session, at a rate the site tolerates.
  4. Solve again only when a challenge reappears, and start the next session on a fresh IP.

What keeps it working

One solve covers many pages

A cleared session usually serves many requests. Budget solves per session, not per page.

Keep the session coherent

Same proxy, same user agent, same TLS fingerprint for the whole run. Most “the cookies do not work” reports are a mismatch here.

Cap spend per job

Give each crawler its own API key with a daily limit, so a loop that solves on every request cannot drain the balance.

Questions

How many solves does a crawl need?

Usually one per session, not one per page. A cleared session keeps working until the site challenges it again, which depends on the site and on your request rate.

Do I need a browser for the product pages?

No. Once the session is cleared, product pages are ordinary HTTPS requests from a client with a Chrome TLS fingerprint.

Which sites can I monitor?

Each domain is approved once before it can be solved, usually within a day. Request the retailers you need from the dashboard.

Solvers

Page updated .

Try it on your own target.

Login, copy a key, send one request. If the solve fails, it costs you nothing.

Price monitoring on protected retail sites | notabot