Why scrapers rot: build vs buy when you scrape hundreds of shops

Scraping one e-shop once is genuinely easy, and we won't pretend otherwise. An afternoon, a hundred lines of Python, done. The trouble is that nobody actually wants data from one shop once. Price intelligence, market analytics, feeds for an AI agent: every real use case means many shops, refreshed on a schedule, indefinitely. And the moment you sign up for that, you stop writing a scraper and start running a fleet.

In short: scraping one shop once is easy; keeping hundreds of shops correct every day is an operations job. At 99% daily reliability per shop, 300 shops mean about three failures every day. In our monitoring of 950 shops, 92 produced at least one data-corrupting change within five days, and the most expensive failures throw no error at all. Build it yourself for a handful of stable sites; buy when the fleet grows.

Play it out

Here is a 30-shop pipeline on a forgiving timeline. Every breakage type below is drawn from our production monitoring, and the specific cases are real, anonymized incidents. Your job is simple: keep clicking, keep fixing.

Why it never stops breaking

None of this is exotic. Shops aren't trying to break your scraper; they're just alive:

• Markup drift. Redesigns, A/B tests, new sale badges, a CSS class renamed by someone's framework upgrade. Your selectors were written for a page that no longer exists.
• Rendering moves client-side. The price that used to sit in the HTML now arrives via JavaScript. Plain HTTP scraping quietly returns nothing; you now maintain a browser farm.
• Anti-bot escalation. Rate limits, fingerprinting, challenge pages, IP reputation. In our production fleet of 2,024 shops, one in four can't be scraped with plain HTTP requests: 13% need a real browser, and another 12% need residential proxies or extra work to keep the page stable. Daily crawling raises the stakes considerably.
• Semantic changes. The page still renders, the scraper still returns numbers, and the numbers are wrong: standard and discounted price swap places, coupon prices appear, a variant's price lands on the wrong product. No exception is ever thrown.

The silent ones are the expensive ones. A crash you notice the same morning. A wrong price flows into your repricing engine, your market report, or your agent's answer, and you find out from an angry customer weeks later. In our five-day production sample, the most frequently broken fields were exactly the ones money depends on: standard price, discounted price, coupon price, availability.

The math is not on your side

Say a single shop's scrape is 99% reliable on any given day, which is generous. One shop breaks about once a quarter: annoying, manageable. Now run 300 shops daily at that same reliability. That's three expected failures every single day, forever. There is no week in which everything works. Reliability per shop doesn't transfer to the fleet; the fleet inherits the failures of every member, every day.

Our own numbers, from the monitoring layer that watches data quality across 950 continuously scraped shops: in one five-day window, 92 shops produced at least one data-corrupting change, 186 incidents in total. About half were repaired automatically by the healing layer without a human ever looking. For comparison, across 600,000+ scrape runs, outright crashes are around 0.1%. The work isn't in the crashes. The work is in everything that doesn't crash.

What running it properly actually takes

This is the iceberg under “we'll just scrape it ourselves”:

• Proxy management: residential pools, rotation, per-shop routing, reputation hygiene. A line item and a part-time job.
• Browser rendering at scale: headless fleets for the shops that need them, with their own memory leaks, timeouts and version churn.
• Data validation: something has to check every day that the numbers are not just present but plausible, or the silent failures walk straight into your database.
• Self-healing: when markup drifts, someone or something has to re-learn the page. Doing this by hand across hundreds of shops is a full-time role; our engine saves a new extraction rule automatically for about half of all incidents.
• Monitoring and alerting: per-shop health, per field, with history, or you learn about breakage from your customers.

Build vs buy: when DIY is the right call

Honestly: sometimes. If you need a handful of stable sites, scraped occasionally, with a developer on hand who treats maintenance as acceptable overhead, build it yourself and skip the vendor. The same goes for one-off research pulls and for learning projects, which are a genuinely great way to understand this problem. The calculus flips when the fleet grows past what one person patches on Friday afternoons, when the data feeds something that costs money when wrong, or when “mostly fresh” stops being good enough.

The build-vs-buy question, stated precisely: not “can you write a scraper?” (you can), but “do you want to staff an operations function whose entire job is keeping other people's websites readable?” That function is our product, DeepScout (deepscout.ai): a managed e-commerce scraping service. It already runs across 1,900+ shops and 166M+ products, exposed as an API and an MCP server for Claude and other AI agents, with the healing, validation and matching layers included. Usage-based pricing, from €0.40 per 1,000 products.

Read next: Product matching without EAN or GTIN: why it's the hard part. Once the data flows, the next problem is telling which listings across shops are the same product.