Skip to main content

What this does

Searchable’s LLM Analytics dashboard shows you, in near real-time, which AI assistants — ChatGPT, Claude, Perplexity, Google’s AI Overviews, and the rest — are crawling your site, what pages they’re hitting, and which of those crawls turn into referred users. To do that, Searchable needs to see your inbound HTTP requests. The setup below points your hosting platform’s request logs at our ingest endpoint. For log and edge integrations, we classify the AI bots and drop everything else. The section on what gets sent, further down this page, summarises what is stored.
Setup takes about 5 minutes. There are no agents to install and no code to change in your application.
Would rather not test this against production first? Register a staging site, follow your platform’s guide against that, and confirm events arrive before you touch production. Staging traffic is reported separately and never counts toward your production numbers.

Pick your platform

Vercel

One-click Log Drain. Works on Pro and Enterprise.

Cloudflare Worker

Edge Worker — works on any Cloudflare plan, including Free.

Cloudflare Logpush

Native Logpush job — Enterprise plan only.

Amazon CloudFront

Standard Logging V2 via Firehose — any CloudFront distribution.

Netlify Edge Function

Edge Function — works on every Netlify plan, including Free.

Netlify Log Drain

HTTP Log Drain — Enterprise plan only.

Fastly

Real-Time Log Streaming via HTTPS endpoint.

Akamai

DataStream 2 HTTPS push — any property with DataStream 2 entitled.

Google Cloud Platform

HTTPS Load Balancer / Cloud CDN logs via Pub/Sub push.

WordPress

Must-use plugin — captures AI crawlers at the origin on any self-hosted site.

Other platforms

Custom REST API or middleware for everything else.

Which one should I use?

Use Vercel Log Drain. It’s the most direct path — Vercel streams request logs to Searchable with one entry in your project settings. No code changes.
Use the Cloudflare Worker. It works on every Cloudflare plan including Free, and the Free tier easily covers ~100k requests/day. The Worker forwards AI-bot traffic to Searchable from the edge — no latency added to real users.
You can use either path. Cloudflare Logpush is the most “native” option — Cloudflare batches and ships logs directly with no Worker involved. If you’d rather avoid the Enterprise-only Logpush feature, the Worker path works on Enterprise too.
Use Amazon CloudFront → Firehose. CloudFront’s free Standard Logging V2 feature ships records to Amazon Data Firehose, which forwards them to Searchable. Works on any CloudFront distribution — you only pay for the Firehose ingestion.
Use the Netlify Edge Function on any plan (including Free), or Netlify Log Drain if you’re on the Enterprise plan. The Edge Function path runs at Netlify’s edge with no added latency for real users.
Use Fastly Real-Time Log Streaming. Configure an HTTPS Logging endpoint inside the Fastly UI — no code changes — and Fastly streams request logs to Searchable.
Use Akamai DataStream 2. Create a DataStream 2 stream in Akamai Control Center, pick the property fronting your domain, and point it at Searchable’s HTTPS endpoint. Requires the DataStream 2 product on your Akamai contract — your account team can confirm if it’s enabled.
Use Google Cloud → Pub/Sub push. Cloud Logging captures every HTTPS Load Balancer or Cloud CDN request; a Pub/Sub topic with a push subscription forwards those logs to Searchable.
Use the WordPress plugin — a must-use plugin that forwards AI-crawler requests from the origin, with no theme changes and no JavaScript. One caveat: if your site is behind Cloudflare, or runs a full-page cache that serves HTML without booting PHP, prefer the Cloudflare Worker — it sits in front of the cache and sees every request.
Use the custom integration. Searchable accepts a small REST POST from your application middleware or any platform that can ship structured request logs.

What gets sent to Searchable

For each AI-bot request, Searchable records:
  • HTTP method, path, and host
  • User agent
  • Referer
  • Country code (geo-IP, no precise location)
  • Client IP address (see below)
  • Response status and bytes
  • Timestamp
Request and response bodies are never sent. The log and edge integrations send no headers other than User-Agent / Referer. Query strings. The Cloudflare Worker drops the query string from the request URL before sending. Every other integration sends the URL as your platform or your code provides it, which can include the query string. The path Searchable stores never includes it, but the full URL is stored as sent, and some integrations (Akamai DataStream 2, Google Cloud, the REST API and application middleware) also store the query as parsed parameters. Client IP addresses. The Cloudflare Worker anonymises the client IP address before sending it by default (the last IPv4 octet is zeroed; an IPv6 address keeps its first 48 bits). Log integrations (Cloudflare Logpush, CloudFront, Akamai DataStream 2, Vercel and the other log drains) and the Netlify Edge Function send the client IP address as your platform reports it, and it is stored with the event. The REST API stores the IP address you send. The WordPress plugin zeroes the last IPv4 octet (or IPv6 group) on your server before sending, so the full IP never leaves your site, and can be set to omit the IP entirely. Requests that aren’t AI bots. Log drains deliver every request your platform logs, and by default the Cloudflare Worker forwards every request after a cold start until its bot registry has loaded, and requests with no user agent after that (its guide shows how to turn both off). Searchable discards requests that don’t match its bot registry on arrival and does not store their path or IP address, with one exception: requests to UCP (Universal Commerce Protocol) endpoints (/.well-known/ucp, /ucp, /api/ucp/* and any others Searchable recognises) are kept whatever their user agent. Custom integrations are different: Searchable stores every event you send through the REST API or your application middleware, human requests included, so filter before sending if you only want AI-bot requests stored. Each platform guide says exactly what leaves your infrastructure.

Verifying the connection

After completing setup, return to LLM Analytics → Setup in your Searchable dashboard. The status strip at the bottom updates as events arrive:
  1. Waiting for first event — your platform hasn’t sent anything yet. AI bots typically hit any indexed site within a few hours.
  2. Connected — events are flowing. The strip shows the count from the last 24 hours.
If the status doesn’t update within a few hours, see the troubleshooting section in the platform-specific guide you followed.

Next steps

Add a domain

LLM Analytics is scoped to your project’s primary domain.

Connect Search Console

Layer in keyword data so you can correlate AI crawls with search demand.