AI Search now supports custom HTTP headers for website crawling, solving a common problem where valuable content behind authentication or access controls could not be indexed. Previously, AI Search could only crawl publicly accessible pages, leaving knowledge bases, documentation, and other protected content out of...
Software Engineering
AI Search - AI Search support for crawling login protected website content
Cloudflare Developer Platform Changelog · 2025-11-19
Related items
Software EngineeringCloudflare Developer Platform Changelog
AI Search - Website Source CSS content selectors for precise content extraction in AI Search
AI Search now supports CSS content selectors for website data sources. You can now define which parts of a crawled page are extracted and indexed by specifying CSS selectors paired with URL glob patterns. Content selectors solve the problem of indexing only relevant content while ignoring navigation, sidebars, foote...
Software EngineeringCloudflare Developer Platform Changelog
AI Search - AI Search path filtering for website and R2 data sources
AI Search now includes path filtering for both website and R2 data sources. You can now control which content gets indexed by defining include and exclude rules for paths. By controlling what gets indexed, you can improve the relevance and quality of your search results. You can also use path filtering to split a si...
Software EngineeringCloudflare Developer Platform Changelog
AI Search - Metadata filtering and multitenancy support in AutoRAG
You can now filter AutoRAG search results by folder and timestamp using metadata filtering to narrow down the scope of your query. This makes it easy to build multitenant experiences where each user can only access their own data. By organizing your content into per-tenant folders and applying a folder filter at que...
Software EngineeringCloudflare Developer Platform Changelog
AI Search - AI Search UI snippets and MCP support
AI Search now supports public endpoints, UI snippets, and MCP, making it easy to add search to your website or connect AI agents. Public endpoints allow you to expose AI Search capabilities without requiring API authentication. To enable public endpoints: Go to AI Search in the Cloudflare dashboard. Go to AI Search...
Software EngineeringCloudflare Developer Platform Changelog
AI Search - Custom metadata filtering for AI Search
AI Search now supports custom metadata filtering, allowing you to define your own metadata fields and filter search results based on attributes like category, version, or any custom field you define. Define a custom metadata schema You can define up to 5 custom metadata fields per AI Search instance. Each field has...
Software EngineeringCloudflare Developer Platform Changelog
Email Routing - Local development support for Email Workers
Email Workers enables developers to programmatically take action on anything that hits their email inbox. If you're building with Email Workers, you can now test the behavior of an Email Worker script, receiving, replying and sending emails in your local environment using wrangler dev . Below is an example that show...