If you want to access data from websites which prevent it, you gotta use a headless browser with Residential Proxy Network Like Bright Data (formerly Luminati).

I&#x27;m curious, how do you deal with Cloudflare and similar anti-bot systems? Just keep shopping the job around to different proxies?

I&#x27;ve been reading up on crawler architecture. The two most useful sources I&#x27;ve found are the blog post &quot;Crawling a billion web pages in just over 24 hours, in 2025&quot; and the Mercator paper (&quot;Mercator: A Scalable, Extensible Web Crawler&quot;).Both of these, and most other material I&#x27;ve come across, focus on crawling the broad open web rather than a targeted set of domains. For product prices it&#x27;s the latter. Mercator calls out DNS resolution as a major bottleneck, for example, but when you&#x27;re only hitting a few hundred domains that isn&#x27;t really a concern.The other gap is that both assume static HTML. For our use case we need a headless browser, and we also have to deal with Cloudflare and similar anti-bot systems.For product prices specifically, a lot of sites publish price feeds which simplifies things, but plenty don&#x27;t, and getting good coverage still requires scraping. Our current system does about 500M pages&#x2F;day and we&#x27;re looking to improve its performance.Does anyone here have experience in this space, or know of articles&#x2F;blog posts on scaling targeted (rather than broad) crawlers with headless browsers? Any pointers appreciated.

Ask HN: Scaling a targeted web crawler beyond 500M pages/day

Comments

faangguyindia

4lx87