When to Use a Proxy for Crawling in BearAudit
Optional proxy rotation can help with large crawls or rate limits. When to enable it, how to set the list URL, and how BearAudit uses your proxy pool.
BearAudit can use a proxy pool when crawling: requests are sent through rotating proxies from a list you provide. Useful when the target site rate-limits by IP or when you're crawling at scale.
When a proxy helps
- Rate limiting — The site throttles or blocks too many requests from one IP. Spreading requests across proxies can keep the crawl moving.
- Large crawls — Thousands of URLs in one run may trigger anti-bot or rate limits. Proxies distribute the load.
- Testing from multiple regions — Less common, but some use proxies to see how the site responds from different locations.
When you don't need one
- Your own site, small or medium — Most of the time a single IP is fine. Enable proxy only if you hit limits or blocks.
- Staging / dev — Usually low volume; proxy adds complexity without benefit.
How to set it up in BearAudit
- Get a proxy list — A URL that returns a list of proxies (one per line or in a format your app supports). Some providers give you a "list URL" that updates.
- Settings → Proxy — Enable proxy and paste the list URL. BearAudit loads the list into a pool.
- Crawl — Start a crawl as usual. Requests go through the pool; if the list is exhausted, BearAudit can refresh from the list URL (e.g. on a schedule or manually).
Proxy is optional. Leave it off if you're not hitting limits. All crawl data is still stored locally; only the HTTP requests use the proxy.