AI Search

How to read crawl-to-referral ratios

See how Cloudflare measured crawls and identifiable referrals, compare three historical platform series, and learn which counts to check when the ratio moves.

Green page requests and blue visitor tokens enter two openings on a comparison instrument

Key takeaways

  • Cloudflare divides crawler requests that returned HTML by HTML requests with a recognized platform referrer. [2]

  • In January–July 2025, the ratios for Google, OpenAI and Perplexity followed different paths. OpenAI peaked in March, while Google and Perplexity peaked in April. [1]

  • A falling ratio can follow fewer crawls, more identifiable referrals or a change in both counts. Check both before explaining the movement. [2]

  • Review cited pages in saved answers separately from crawl and referral activity. Cloudflare’s formula uses two request counts. [2]

A crawl-to-referral ratio tells you how many crawler requests there were for every identifiable referral request in a defined window. If it falls, check which count changed. That distinction matters to a publisher deciding whether to investigate crawler access, review traffic attribution or examine the pages cited in AI answers. [2]

Cloudflare’s January–July 2025 data gives us three examples of how the ratio can move. We use all seven monthly values for each selected platform to show what a chart reveals, what the underlying request counts would add and how to apply the distinction to a website review. [1]

What counts as a crawl or referral?

Cloudflare groups requests by search or AI platform across the traffic its network observes. For the numerator, it counts requests from associated crawler user agents where the response was HTML. For the denominator, it counts requests for HTML where the referrer includes a hostname associated with that platform. Each published ratio expresses the first count per one request in the second. [2]

A value of 20 therefore represents 20 qualifying crawl requests per identifiable referral request in the measured scope. Repeated requests can contribute more than once. A platform may have several crawlers, including agents that fetch pages for different purposes. Cloudflare’s grouping can combine that activity under one name. The Google series in this analysis describes Cloudflare’s platform grouping. A product-specific question about AI Mode would require its own evidence. [2]

How three platforms moved in 2025

The chart sets each platform’s January ratio to 100. Each subsequent point divides the published monthly ratio by that platform’s January value and multiplies by 100. This lets you compare the movement of ratios that started at very different levels. A line at 200 represents twice the platform’s January ratio. Use the underlying table to see the reported levels. [1]

Crawl-to-referral ratios indexed to January 2025

Three selected platform series from Cloudflare Radar’s published January–July 2025 observations.

592.11444.0825296.055148.02750JanFebMarAprMayJunJulMonth in 2025
  • Google
  • OpenAI
  • Perplexity
Full data table
Crawl-to-referral ratios indexed to January 2025 · index, January = 100
Month in 2025GoogleOpenAIPerplexity
Jan100100100
Feb165.79145.76101.28
Mar384.21182.11368.68
Apr592.1198.57551.1
May439.4781.78364.65
Jun344.74136.02367.4
Jul142.1189.65356.78

Each line starts at 100. Ansvra calculated the index from the published monthly ratios. Use the raw-ratio table to compare levels and inspect the request counts separately to explain a change. [1] [2]

OpenAI’s ratio reached its highest reported level in March, at 2,217. Google and Perplexity peaked in April, at 22.5 and 300.9 respectively. By July, Google’s ratio was back near its January level, Perplexity’s was still more than three times its January value, and OpenAI’s was below where it started. The monthly path matters. Google ended July 42.11% higher than January, but its April level was almost six times its January value. [1]

Complete reported ratios for the selected platforms

The table includes all 21 observations used in the analysis. Each number is the reported count of qualifying HTML crawl requests per identifiable HTML referral request. Cloudflare published these monthly ratios in August 2025. The underlying monthly crawl and referral totals, geographic breakdown and per-site results are unavailable in the selected table. [1]

Month (2025)

Google

OpenAI

Perplexity

Jan

3.8

1,217.4

54.6

Feb

6.3

1,774.5

55.3

Mar

14.6

2,217

201.3

Apr

22.5

1,200

300.9

May

16.7

995.6

199.1

Jun

13.1

1,655.9

200.6

Jul

5.4

1,091.4

194.8

January-to-July change

Ansvra calculated each change by dividing the displayed July ratio by the displayed January ratio, subtracting one and multiplying by 100. Values are rounded to two decimal places. The last column preserves Cloudflare’s printed changes, which differ slightly from calculations based on the displayed values. [1]

Platform

January ratio

July ratio

Ansvra change

Cloudflare change

Google

3.8

5.4

+42.11%

+43%

OpenAI

1,217.4

1,091.4

−10.35%

−10.4%

Perplexity

54.6

194.8

+256.78%

+256.7%

Read a relative change alongside its starting level. The 256.78% increase for Perplexity began from a ratio of 54.6. OpenAI’s July ratio remained higher in absolute terms, even after it fell by 10.35%. An indexed chart helps compare paths over time. Use the original values when your question is how far apart those ratios were in a particular month. [1]

Check both counts before explaining a change

Imagine a publisher with 1,000 qualifying crawl requests and 50 identifiable referral requests in one month. The ratio is 20. In the next month, a ratio of 10 could arise from 500 crawls and 50 referrals, or from 1,000 crawls and 100 referrals. It could also occur if both counts went down, such as 400 crawls and 40 referrals. These are arithmetic examples. The same ratio movement would lead to three different investigations.

Keep a row for each comparable reporting window with the two counts next to the quotient. Add the website, platform grouping, content type and any known change in what your systems can observe. If the recorded referral count is zero, the quotient is undefined. If the count was never captured, mark it as unavailable. When referral counts are very small, a difference of just a few requests can substantially move the ratio. Put the counts next to the chart so readers can see the scale of the change.

How to investigate a change in the ratio

A workflow for keeping crawl, referral and answer evidence in their proper scope.

  1. Match scope

    Use the same website, operator grouping, dates and content type before comparing the two counts. [2]

    Next: Count crawls

  2. Count crawls

    Count the matched crawler requests within the chosen scope. Cloudflare’s metric uses requests that returned HTML. [2]

    Next: Count referrals

  3. Count referrals

    Count HTML referral requests with a recognized referrer. Record traffic that cannot be attributed separately. [2]

    Next: Check both counts

  4. Check both counts

    A change in the quotient may reflect movement in either count. Inspect both before deciding what to investigate. [2]

    Next: Review citations

  5. Review citations

    Read saved answers to see which pages were cited. Keep that observation separate from the two traffic measures. [2]

This is an editorial review workflow. The steps organize the evidence needed for the decision. [2]

If crawl requests fell while referrals held steady, begin with the crawler and pages where the requests changed. A change isolated to one agent or section gives you a narrower investigation than a site-wide shift. If referral requests changed, check that the date range, included pages and recognized referrers still match. If both moved, report their directions and sizes individually before describing the quotient. The monthly Cloudflare ratios offer historical context for this workflow. Their aggregate values cannot identify which of these scenarios occurred for a particular publisher. [1] [2]

Account for gaps in referral evidence

The denominator depends on what arrives with a request. In its methodology, Cloudflare describes a case where traffic referred from Claude’s native app lacks the referrer information used for platform attribution. That makes some visits invisible to the denominator. The effect on the quotient depends on how many requests are missing from the count. Cloudflare’s example provides no estimate of the gap for the three series here. [2]

Keep measurement units visible when comparing reports. An analytics session can include several page requests, and its source is assigned using the analytics system’s rules. If your referral evidence consists of sessions and your crawl evidence consists of requests, report both series with their own units. A publisher reproducing the Cloudflare definition would need comparable request-level counts and compatible platform groupings. Cloudflare’s January–July 2025 observations also describe a historical network aggregate. Use your own reporting window to investigate current changes on your website. [1] [2]

Separate trays for page requests and referral tokens beside a bookmarked page card
Keep crawl requests, identifiable referrals and cited pages separate in the review.

Bring crawl, visit and citation evidence into the same review

Start with a concrete question, such as why the help section received less recognizable traffic from an AI platform this month. Use one date range and an explicit group of URLs. Pull three observations into the review, each labeled with its own unit and coverage.

  1. Crawler activity. Review recorded activity by agent, page and response. A team using Ansvra’s AI Agent Analyzer can look for changes concentrated on the pages in the review. AI Agent Analyzer

  2. Recognizable visits. Review the sources and landing pages in connected analytics. Write down what the source grouping can identify, and keep its visit or session count separate from request counts. Traffic from AI

  3. Cited URLs. Read saved answers for the questions those pages cover. Record which URLs are referenced and which prompts produced them. That helps you choose pages for a closer content review. Citation Source Analysis

Say a help page appears in saved answers in both periods, while recognizable visits to it fall. Start by reviewing the landing-page traffic and how the platform was identified. If the page disappears from otherwise comparable saved answers, look at the URLs cited in its place and how they answer the prompt. These observations help you choose the next check. Connecting an individual crawl, answer and visit would require evidence linking those events.

Record the change you can verify

In a monthly review, write one sentence about each measure before writing a conclusion. For example, a team might find that recorded crawler requests fell on a set of help pages, that recognizable visits to those pages were stable, and that the same pages still appeared in its saved answers. That pattern gives the team a reason to inspect the crawler activity first. It also gives the next reviewer three observations to compare after the investigation.

Capture the coverage alongside each observation. If crawler recording started partway through the first month, use two complete windows or flag the incomplete period. If the team changed its referrer grouping, document the date of the change and apply a consistent grouping to the periods it compares where the original evidence permits. For saved answers, keep the prompts and the platform consistent when reviewing changes. These checks help distinguish an observed change on the site from a change in how the team recorded it.

Close the review with the next question and the evidence needed to answer it. A crawler investigation might need a page-level breakdown of responses. A referral investigation might need a comparison of recognized sources on the same landing pages. A content review might need the cited pages and the saved answer text. Assign the next check to the team that can inspect that evidence.

Connect Cloudflare and see what AI crawlers do on your site

Cloudflare’s published ratios show how platforms moved across its network. Which AI crawlers are reaching the pages that matter to your business, and how is your site responding? Ansvra gives you a way to find out.

If your site is on Cloudflare, Ansvra guides you through the connection. Open AI Agent Analyzer in your project, select Connect a source, and choose Cloudflare. Once activity comes in, you can measure requests by crawler and page, compare periods, and find pages where requests failed.

After refreshing a product guide, check whether crawlers are reaching it and whether your site is serving it successfully. Then review whether the guide appears in saved AI answers and whether identifiable AI visits to it changed. Together, those separate signals help you decide where to spend your time next, from investigating access to improving the page or checking how visits are attributed.

See how to connect Cloudflare in Ansvra.

Summary

Use Cloudflare’s historical ratios to understand the relationship between two measured request counts. For your own review, match the website, dates, platform and content type, then check crawls and identifiable referrals separately. Read saved answers alongside those counts to see which pages were cited and decide where to investigate next.

Sources

  1. Cloudflare’s The crawl-to-click gap: Cloudflare data on AI bots, training, and referralsAccessed
  2. Cloudflare’s The crawl before the fall… of referrals: understanding AI’s impact on content providersAccessed

Discussion

Join the discussion

Start the conversation

Share a question or tell us what worked for you.

Comments appear after review. How we handle comments

The Ansvra journal

Keep reading

Browse all articles

Ansvra

A clearer next step.

MEET YOUR AI GUIDE

A little clarity.
A clearer next move.

Ask about Ansvra, AI visibility, or your next step with content.

Opening chat…