Follow the crawlers.
Find the pages
they reach.
See which AI crawlers request your pages, how your website responds, and where access deserves a closer look.
Bring website access into your AI visibility research, with a clear distinction between crawler visits, answer citations, and human traffic.
Activity reports use a supported website connection or uploaded logs. Request allowances and detailed history vary by plan.
Your website
Pages, responses,
and access settings.
Before a page can be useful,
understand its access.
AI Agent Analyzer shows recorded crawler requests, the pages they targeted, and how your website responded. See how activity changes over time and find pages with failed requests.
Use saved answers to see which pages an AI response cited.
Same website.
Different responses.
Select a page to explore what its response tells you, and what your team could investigate next.
A recorded response and an access-policy check answer different questions. Read both in context.
The website returned the page.
A successful response tells you the website served the requested page. Review the page and request details, then investigate its content separately.
The request was sent elsewhere.
A redirect response points the request toward another destination. Investigate the redirect and destination before treating this as a successful visit to the content you intended.
The page was not made available.
An access-denied response shows that the website restricted this request. Review whether the restriction is intentional and inspect the relevant page access settings.
These response types explain page access. A successful response does not prove that the page was indexed, cited, or read by a person.
Connect the website.
Read the activity.
Follow up on the page.
Open AI Agent Analyzer in your project to set up reporting and review your website’s access rules.
Connect or import
Connect a supported website for ongoing reporting, or upload saved logs to review activity from an earlier period.
Explore the observed visits
Choose a date range in Overview. Open Pages or Bots to drill into recorded requests, filter by response status, and investigate failures.
Investigate page access
Open Access to see what your robots.txt rules say for each bot. Use the URL tester to check the response and final URL for a specific page.
Uploads provide saved history. An ongoing connection may be waiting for its first delivery, and the activity report reflects the crawler requests recorded in the data received.
A visit.
A citation.
A person.
Each belongs in your research, but they measure different events. Choose a signal to see what it can tell you.
Did a crawler request a page?
Observed website requests show crawler activity and the response your website returned. They help you investigate page access.
Did an AI answer reference the page?
Saved answer evidence shows the pages cited by a particular response. Open the answer and its sources to understand the reference.
Did a person arrive at the website?
Referral reporting describes recorded website visits. It answers a different question from whether a crawler requested a page or an AI answer cited it.
Clearer observations.
Better questions to ask next.
A crawler report helps you investigate access. Pair it with your page knowledge and saved answer evidence to understand the bigger picture.
Explore the answer behind a citationRead crawler identity carefully
Recognized request identities help classify activity. A crawler label does not independently verify the operator or establish which AI model used the page.
Keep coverage with the conclusion
Missing observations do not prove that no visit occurred. Check the connection, available dates, and reporting limits before reading a gap as inactivity.
Use access checks as evidence
An access rule and a recorded page response are different observations. A rule permitting access does not prove that a crawler visited or that a page was indexed.
Website access,
crawler visits,
and what comes next.
How the reports work and how to interpret the activity you see.
How do I get started?
Choose your website project and open AI Agent Analyzer. An account administrator can connect a supported website, and members with editing access can upload saved logs or run access checks.
Can I check page access before connecting my website?
Yes. When access checks are available to your account, use the Access tab to review robots.txt or test a page URL. The activity reports need recorded requests from a connection or an upload.
Does a crawler visit mean my page was cited?
No. A request shows that a crawler requested a page. A citation is a reference in a saved AI answer. Review each event in its own report.
Does a visit prove that my content was used for training?
No. A recorded request does not establish training use, indexing, or which model may have used the content.
Does a connected website guarantee activity?
No. Connection status, received information, and actual crawler observations are separate. Check available dates, connection status, and coverage before interpreting the report.
How are access rules different from page responses?
Access rules describe the policy that applies to a crawler. A recorded response describes what the website returned for a particular request. An allowed rule does not prove a visit, and a successful response does not prove indexing or citation.
Does no recorded activity mean no crawlers visited?
No. Missing observations can reflect reporting coverage, dates, connection state, or unavailable information. Keep gaps separate from a confirmed record of activity.
Can I upload saved logs instead of connecting my website?
Yes. Upload a log file to review the recorded crawler activity for the dates it covers. Connect a supported website to receive new activity as it becomes available.
Does this include human website visits?
Crawler requests are separate from human referral traffic. Use the appropriate website traffic reports for recorded audience visits, and saved answer reports for citations.
How much request history can I see?
Request allowances and the time you can view individual requests vary by plan. Check the available dates and detailed-history window in your report.
Know the paths to your pages.
Bring crawler activity into your research, with every signal in its proper context.