Skip to content

AI Search - AI Search support for crawling login protected website content

UpdateUnverifiedAdded Sep 24, 2026

AI Search now supports custom HTTP headers for website crawling, solving a common problem where valuable content behind authentication or access controls could not be indexed. Previously, AI Search could only crawl publicly accessible pages, leaving knowledge bases, documentation, and other protected content out of your search results.

Topics: Security, Developer tools

Cloudflare's release notes

More Cloudflare developer platform releases

Every Cloudflare developer platform release

Also shipped on Nov 19, 2025

Cloudflare in November 2025

Sources: each vendor's own release notes, changelogs and GitHub releases, read daily to monthly by how often it posts. Logos via logo.dev; trademarks belong to their owners.

New releases by email

Tuesday mornings: the week's data, AI and developer-tools releases, only in weeks when something shipped.

Double opt-in. Unsubscribe any time.