Angaben zum Datum
Datum aus der Quelle.
Erstmals gesehen am .
Developer Platform von Cloudflare
Browser Run: Ganze Websites per /crawl-Endpoint crawlen
Der neue /crawl-Endpoint von Browser Rendering (Open Beta) durchsucht ab einer Start-URL automatisch eine ganze Website, rendert die Seiten und liefert sie asynchron als HTML, Markdown oder strukturiertes JSON, wobei robots.txt und AI Crawl Control standardmäßig beachtet werden.
Edit: this post has been edited to clarify crawling behavior with respect to site guidance.
You can now crawl an entire website with a single API call using Browser Rendering's new /crawl endpoint, available in open beta. Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON. The endpoint is a verified bot (intermediary agent) that respects robots.txt and AI Crawl Control ↗︎ by default, making it easy for developers to comply with website rules, and making it less likely for crawlers to ignore web-owner guidance. This is great for training models, building RAG pipelines, and researching or monitoring content across a site.
Crawl jobs run asynchronously. You submit a URL, receive a job ID, and check back for results as pages are processed.
# Initiate a crawl
curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
-H 'Authorization: Bearer <apiToken>' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://blog.cloudflare.com/"
}'
# Check results …