Scraping online stores behind Cloudflare without getting blocked: how common it is, what works, what it costs
We checked 1,151 online stores we scrape. A third sit behind Cloudflare, one in six is hard to scrape, and on .com stores it's one in three. Here's what that means for success rates, speed and price, with real runs.
Cloudflare, Akamai, DataDome and similar services sit in front of many online stores. For a scraper, that means challenge pages instead of products, 403 errors, and a script that works on Monday and stops on Thursday.
Most articles on this topic explain how to get past one specific protection. This one answers the questions you ask before that: how many stores are protected, which tools still return products from them, and what it costs per product.
The short answer
- A third of online stores sit behind Cloudflare. Of 1,151 stores we scrape, 34% answer through Cloudflare. A plain request was turned away by 13% of them, and Cloudflare sent 88% of those blocks.
- Most stores aren't hard, but many are. 65% need no browser, 18% need a browser, and 17% have strong protection. On .com stores, 35% are in the hard group.
- A 200 response isn't a product. Protected stores often answer with a challenge page and a success status. Count products with a name and a price, not responses.
- Protection changes the price and the speed, not only the success rate. On DeepScout, a hard store costs €1.20 per 1,000 products instead of €0.40, and a run takes longer.
- Test on your stores before you pick a tool. In our benchmark, the API that was best on a protected store returned nothing from an unprotected one.
Need products from a protected store?
Add the store in the console, or ask Claude through our MCP server. We handle the browsers, proxies and parsing. The free analysis shows how hard the store is, how many products it has and the exact price before you pay.
How many online stores are behind Cloudflare?
We sent one ordinary request to the home page of each of the 1,151 stores we scrape and read the response headers. Cloudflare and some other protection services mark their responses, so this shows who sits in front of a store. It doesn't show how strict their settings are.
Cloudflare in front of 1,151 online stores
| Result | Stores | Share |
|---|---|---|
| Answer through Cloudflare | 390 | 34% |
| Turned away a plain request (403) | 146 | 13% |
| …of which the block came from Cloudflare | 129 | 88% of blocks |
| Answer through Cloudflare, .com stores only | 34 of 82 | 41% |
Checked on 9 November 2026 with one request per store, no browser. 9 stores didn't answer and are left out.
What this means for scraping:
- Cloudflare in front of a store doesn't mean you can't scrape it. Many stores use Cloudflare as a fast CDN with light settings. Others turn on bot checks that block a plain request at once. You can't tell which from the outside until you try.
- Most blocks you'll hit come from Cloudflare. If your scraper gets a 403 or a "checking your browser" page from an online store, it's most likely Cloudflare.
- Akamai, DataDome and others are less common and harder to spot from headers. REI, the Akamai-protected store in our benchmark below, is one of them.
How hard are online stores to scrape?
Every store we scrape has a difficulty: easy, medium or hard. It decides the price, and you see it before you pay.
Difficulty of 1,151 online stores, November 2026
| Difficulty | What it means | All stores | .com stores |
|---|---|---|---|
| Easy | The store gives its pages without a browser | 65% | 46% |
| Medium | A scraper must use a browser to read the pages | 18% | 19% |
| Hard | The store has strong protection | 17% | 35% |
These are stores we scrape successfully, so the numbers leave out stores nobody can scrape. The .com group is 84 stores. The difficulty descriptions are the ones in our pricing docs.
Which scraping APIs still got products from a protected store
In November we sent the same product URLs through eight scraping APIs: 98 from REI, a large outdoor store with Akamai bot protection, and 100 from Brooklinen, a Shopify store with no bot protection. Success means a product with a name and a price.
Products collected: a protected and an unprotected store
| API (mode) | REI (Akamai) | Brooklinen (no protection) |
|---|---|---|
| ScraperAPI | 97/98 | 0/100 |
| DeepScout | 95/98 | 99/100 |
| Zyte | 95/98 | 92/100 |
| Oxylabs (JS rendering) | 82/98 | 85/100 |
| Bright Data Web Unlocker | 59/98 | 0/100 |
| Firecrawl (JSON) | 34/98 | 92/100 |
| Apify E-commerce Scraping Tool | 24/98 | 98/100 |
| ScrapingBee | 0/98 (every mode up to stealth) | 23/50 (AI mode, 50 URLs) |
Three things stood out:
- The store decides the winner. ScraperAPI was the best and the cheapest on REI, at $0.94 per 1,000 products against DeepScout's $1.34. It returned no products from Brooklinen, because Brooklinen's pages carry no schema.org product data to parse. DeepScout was the only API in the top two on both stores.
- Protection splits the field. Apify and Firecrawl collected over 90% of Brooklinen and a third or less of REI.
- Empty successes. Oxylabs' default mode returned HTTP 200 and "parse success" on REI with every field empty. It had parsed the bot challenge page.
Full method, all eight APIs and the raw results: Best e-commerce scraping APIs in 2026.
What a protected store looks like in a real run
In November we refreshed prices for 700 product URLs on each of three UK hi-fi stores. One of them, Richer Sounds, is in the hard group.
700 product URLs per store
| Store | Difficulty | URLs with a product | Time | Cost |
|---|---|---|---|---|
| Sevenoaks | Easy | 694/700 | 30 min | €0.49 |
| Peter Tyson | Easy | 693/700 | 30 min | €0.42 |
| Richer Sounds | Hard | 651/700 | 2 h | €0.78 |
The easy stores returned a product for 99% of the URLs. The hard store returned 93%, took four times longer and cost more per product. For the 35 URLs it couldn't read, the per-URL result says so, and those weren't charged. Peter Tyson's cost covers 603 products, because several URLs led to the same product.
Build or buy: your own scraper, an unblocker or a product API
Getting past protection yourself usually means headless browsers, rotating proxies and solving challenges. That can work. The cost is the upkeep: protection vendors update their checks, stores change their pages, and a scraper that worked last month returns challenge pages today. That's the usual answer to "why do my scrapers keep breaking".
Three ways to get product data from protected stores
| Approach | You get | You maintain | You pay per |
|---|---|---|---|
| Your own scraper | Anything you can parse | Browsers, proxies, parser, fixes | Server and proxy traffic |
| Unblocker API (ScraperAPI, Bright Data, ZenRows) | Raw HTML | The parser | Request, more for JS and premium proxies |
| Product API (DeepScout, Zyte) | Product fields: name, price, stock, EAN | Nothing per store | Product |
With DeepScout you don't pick a mode or a proxy. The free analysis of a store tells you its difficulty and the price before you pay, and you pay only for products collected.
How it works
- 1Analyze · freeAdd the store by URL.You get the product count, the difficulty, the price and a sample of real products. The analysis is free for 10 stores a month.
- 2ScrapeApprove the quote.Easy stores cost €0.40 per 1,000 products, medium €0.70, hard €1.20. You pay for the products collected.
- 3Get the dataDownload or ask your agent.JSON Lines with the same fields for every store, or the same data in Claude or Codex through MCP.
Try it now
Create a project, paste the store URL and wait for the analysis. You see the difficulty, the product count and the price before you pay.
Open the consoleQuestions
Can you scrape stores behind Cloudflare?
A third of the stores we scrape sit behind Cloudflare, so yes, routinely. Some are easy and some are hard. The free analysis tells you which before you pay.
Which protections do you handle?
We don't publish a list. The free analysis tries the store and tells you its difficulty and price before you pay anything.
Can you scrape any store?
No tool can promise that. If the analysis can't read a store, it fails and costs nothing. Marketplaces like Amazon, eBay and Allegro aren't supported.
Why does a protected store cost more?
It takes more work per page. Our price follows the difficulty: €0.40, €0.70 or €1.20 per 1,000 products.
What happens to URLs that fail?
In a URL scrape every URL gets a result, for example OK, NOT_FOUND or IGNORED_RESPONSE. You pay for products with a result of OK or PRICE_NOT_FOUND.
Is scraping protected stores legal?
It depends on the country, the data and how you use it. Public product data like prices is the common case, but terms of service, database rights and personal data rules differ. By default our scraper follows each store's robots.txt. Ask a lawyer about your specific use.
Can my AI agent scrape protected stores?
Yes, through the DeepScout MCP server. The agent asks for the data, the protection is handled on our side, and the agent gets product fields back.
Read next
- Guide: Best e-commerce scraping APIs in 2026
- Use case: Keep prices fresh for product URLs
- Use case: Export a store's full catalog
Get started with the API
Scrape any e-shop into clean product JSON, then match it against the shops you pick. No EAN or barcode required. Usage-based, from €0.40 per 1,000 products.