AI Bot Funnel Analysis: How to Map Crawl Paths in Server Logs and Fix Your Agentic Search Gaps

You know which blog posts rank. You know which pages convert. What you probably don’t know: which sections of your site AI systems have explored thoroughly enough to actually recommend you, and which they’ve barely touched.

That gap isn’t in GA4. It isn’t in Google Search Console. It isn’t in any AI brand visibility dashboard. It lives in one place: your server access logs.

I’ve been doing log file analyses for SEO for years, and I think that the logs have never been more interesting than they are right now.

Server logs record every HTTP request from every bot: training crawlers, search indexers, and the on-demand fetchers that fire when a real user asks an AI a question about your category.

As of May 2026, Googlebot holds approximately 27% of global AI crawler traffic, GPTBot around 10%, Meta-ExternalAgent around 12%, and ClaudeBot has surged from 9% to 20%, now a top-three crawler that most SEO teams aren’t tracking yet, according to Cloudflare Radar.

These shares shift month to month. The Radar dashboard is the live source, but the directional point is stable: AI crawlers are a primary layer of your web traffic, and most teams are asking, “How much AI bot traffic am I getting?”

That’s the wrong question.

Raw bot volume is the first-order question. “Where are they going?” is the one that matters.

A bot that hits your /blog/ section 4,200 times a day while never reaching your /pricing/ page is not a well-informed AI advocate for your product. It’s a well-informed AI expert on your content. Those are not the same thing, and the difference becomes very visible when AI agents start making vendor shortlist decisions on behalf of real buyers.

The difference from traditional log analysis isn’t the data source. It’s the objective.

Googlebot crawls to decide whether your page should rank for a query. AI training crawlers crawl your company’s operations to build a working model.

The consequence of missing coverage isn’t a page not ranking. It’s an AI system that doesn’t know what your product costs, who it serves, or how it differs from alternatives, and therefore can’t include you on a vendor shortlist even when you’d be the right answer.

This article introduces what I call the AI bot funnel model: a method for reading server log data by site section, from awareness-stage content through evaluation and decision pages, to map exactly where your AI visibility is strong, where it thins out, and where it disappears entirely.

I’m assuming you already know what agentic search is (if not, the agentic search guide covers it) and that you can pull logs from your server or CDN (the log file analysis guide covers extraction). What this adds is the analytical layer between extraction and optimization decisions.

What AI bot funnel analysis is

AI bot funnel analysis is the practice of segmenting server log data by site section (blog, features, pricing, docs) and interpreting AI bot crawl patterns as a buyer-journey funnel to identify where AI systems’ knowledge of your site ends.

Log analysis has three layers. Most teams stop at layer 1. This article is about layer 2.

Instead of treating a raw hit count as the output, funnel analysis maps which stages of your content architecture AI bots have actually explored, and surfaces the gaps between what AI systems know about your category expertise versus what they know about your product.

It’s meaningfully different from standard log file analysis in SEO, and it covers two distinct layers that most teams conflate.

Traditional log analysis asks: Is Googlebot crawling this page? Is crawl budget being wasted? Are 404s blocking the index?

The first layer of AI bot funnel analysis asks: Does this AI system know enough about my solutions and pricing to put me on a vendor shortlist?

The second layer goes further: Can an AI agent actually complete a task on my site? Can it book a demo, initiate a trial, or step through a purchase flow on behalf of a user who asked it to?

A site can be thoroughly crawled and well-understood by an AI system and still be completely unusable by an AI agent if its conversion flows require JavaScript interactions, multi-step authentication, or dynamic elements that agents can’t navigate.

These are different failure modes with different fixes, and both show up in your logs if you know what to look for.

The underlying logic isn’t new, it’s just been applied to a different problem. The SEO industry spent years optimizing crawl access to conversion pages because Googlebot needed to see them to rank them.

The same logic applies to AI systems, with greater stakes at play: an AI agent making a vendor recommendation draws on whatever knowledge it has about your site. If that knowledge is limited to your blog posts, the recommendation reflects that. And if the agent is instructed to go further and actually complete a transaction, a site that isn’t agent-accessible doesn’t just lose the recommendation. It loses the conversion.

The AI visibility funnel: TOFU, MOFU, BOFU through a bot’s eyes

The AI visibility funnel maps how thoroughly AI systems have crawled each stage of your buyer journey. Each stage has a distinct content type, a distinct set of AI bots that gravitate toward it, and a distinct consequence for your agentic search visibility when it’s under-crawled.

The following map illustrates crawl paths based on observed behavior across the sources cited in this article. The paths diverge sharply at the MOFU layer and reach near-zero for scheduled crawlers on the Pricing node.

Shadow crawlers are the exception: they arrive at Pricing directly when a real user asks an AI a vendor-evaluation question, regardless of what training crawlers have covered. That asymmetry is what makes the Pricing node the diagnostic focal point of funnel analysis.

TOFU: top of funnel (/blog/, /resources/, /guides/, /learn/)

Top-of-funnel sections are where training crawlers live. GPTBot, the most aggressive AI bot in production, averaging 4,200 hits per site per day across a 30-day study of twelve production sites (DigitalApplied, April 2026), operates breadth-first and strongly prefers /blog/, /docs/, and /about/ paths. If your site has a content section, it almost certainly has strong GPTBot coverage.

Strong TOFU AI coverage means AI systems know your category expertise. They’ll cite your content when answering questions about your topic. What it does not mean: they can recommend you as a vendor for a specific business problem.

Category knowledge is not product knowledge. This is a distinction most teams miss entirely.

This behavior shows up in citation data, too. Similarweb’s Analysis of ChatGPT citation patterns in the US found that over 60% of citations come from pages at folder depth 2 or 3, exactly the shallow, well-linked content that breadth-first crawlers like GPTBot prioritize.

If your most important pages sit deeper in your site architecture, they are less likely to be crawled and cited.

MOFU: middle of funnel (/features/, /solutions/, /use-cases/, /integrations/)

MOFU pages are what tell AI systems: this product solves this problem for this type of customer. They’re consistently under-crawled relative to TOFU because they lack the content density, internal link authority, and update frequency that pulls GPTBot to /blog/.

ClaudeBot is different. It crawls depth-first (average session depth of 5.2 versus GPTBot’s 3.8, per the same DigitalApplied study) and gravitates toward /docs/ and /api/ sections on B2B SaaS sites.

If your product docs are well-structured, you likely have better ClaudeBot coverage on MOFU-adjacent content than your GPTBot logs suggest.

BOFU: bottom of funnel (/pricing/, /demo/, /comparison/, /vs/)

BOFU is where the stakes of agentic search are highest, and where the two layers of AI bot analysis diverge most sharply.

For the knowledge layer: an AI agent helping a user evaluate vendors’ needs, access to pricing context, feature comparisons, and demo or trial information to form a recommendation.

For the action layer: an AI agent instructed to book a demo or start a trial needs to actually navigate and complete those flows.

A BOFU page that is crawled but JavaScript-dependent fails both layers simultaneously. It can’t be read by training crawlers, and it can’t be operated by execution agents. These are also the pages with the lowest AI bot hit rates across the board.

GPTBot’s stated preference for /blog/, /docs/, and /about/ paths means /pricing/ and /demo/ receive materially less coverage, not by accident, but as a direct consequence of breadth-first crawling from a site’s most-linked entry points, which are rarely the commercial decision pages. Per the same study, flat BOFU hit counts in your funnel matrix are the expected output of current AI crawler behavior.

That doesn’t make them acceptable. It makes them a known, fixable problem.

Infrastructure (/docs/, /api/, /developers/)

Technical documentation is ClaudeBot’s territory. Depth-first crawling plus well-structured /docs/ hierarchies means product docs often have the best AI crawl coverage on the entire site. For developer-facing products, that’s a genuine advantage: AI systems handling implementation queries draw heavily on /docs/ content.

The funnel coverage matrix

How to map AI bot crawl paths by site section: the funnel matrix method

The funnel matrix method takes your raw server log data and reorganizes it around a single question: which AI bots are reaching which sections of your site, and how often? Instead of looking at individual URLs, you group hits by section prefix, /blog, /pricing, /docs, and cross-reference by bot type. The result is a matrix that shows your AI visibility funnel in one view.

Most log analysis tools let you filter and group by user-agent and URL path. If you’re working from a raw access.log file, these two commands do the work. Run them in your server terminal or wherever you process log exports.

For funnel analysis, you need three fields:

1. URL path, the
2. User-agent string
3. response code

Everything else is noise for this analysis.

Step 1: Filter for AI bot user-agents

Before you can segment by funnel stage, you need a clean extract that contains only AI bot traffic. This grep command filters your raw access log down to requests from the eight user-agent strings that cover every major AI crawler family: scheduled training crawlers, live search-retrieval bots, and shadow crawlers that fire on real user queries. The output file becomes the input for every subsequent step, so run this first and keep the filtered log intact for reuse.

grep -E "GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Google-Extended|ChatGPT-User|Perplexity-User|Claude-Web" access.log > ai_bot_hits.log

Step 2: Extract section-level path prefix and bot identity

With your filtered logs in hand, this awk command does two things at once:

First, it extracts the section-level path prefix from field 7 of each log line: the first path segment after the domain, such as /blog, /pricing, or /docs. This is what lets you group thousands of individual page hits into meaningful funnel stages rather than counting URLs one by one.

Second, it identifies the bot type from the user-agent string and tags each hit accordingly.

The final sort and uniq -c pipeline aggregates the result into a count-sorted list showing how many times each bot hit each section.

Here is the full command:

awk '{

n = split($7, parts, "/")

section = (n >= 3) ? "/" parts[2] : "/"

if ($0 ~ /GPTBot/) bot = "GPTBot"

else if ($0 ~ /ClaudeBot|Claude-Web/) bot = "ClaudeBot"

else if ($0 ~ /PerplexityBot/) bot = "PerplexityBot"

else if ($0 ~ /ChatGPT-User/) bot = "ChatGPT-User"

else if ($0 ~ /Perplexity-User/) bot = "Perplexity-User"

else if ($0 ~ /OAI-SearchBot/) bot = "OAI-SearchBot"

else if ($0 ~ /Google-Extended/) bot = "Google-Extended"

else bot = "Other"

print section "\t" bot

}' ai_bot_hits.log | sort | uniq -c | sort -rn | head -40

Step 3: Build the funnel matrix

The output you get on step 2  is a count-sorted list showing how many times each bot hit each section of your site. Copy those numbers into a spreadsheet with site sections as rows and bot types as columns. That is your funnel matrix. Any cell with zero or near-zero hits is a funnel gap.

What the matrix tells you that raw counts don’t

A site with 50,000 GPTBot hits per month sounds well-crawled. If 49,500 of those hits are on /blog/ and 500 are spread across the rest of the site, that site has a severe MOFU/BOFU funnel gap. AI systems know their content library but have minimal exposure to their product or commercial pages.

You’d never see that in an aggregate bot traffic report. It jumps out immediately in the funnel matrix.

What your funnel gaps are telling you

A funnel gap is not a crawl budget problem. Before you go optimizing for bot volume, read what each gap pattern is actually signaling.

The TOFU trap

The most common pattern I see across B2B SaaS sites: strong GPTBot coverage on /blog/, weak coverage on /features/ and /solutions/, near-zero on /pricing/.

The consequence is a specific kind of AI invisibility: it doesn’t show up as low brand-mention rates for category queries because the blog content is getting cited. It shows up as an absence from AI-generated vendor shortlists, comparison responses, and decision-stage queries.

An AI agent tasked with finding the best competitive intelligence platform for an enterprise marketing team will not cite a blog post about keyword research methodology as a vendor recommendation. It’ll look for pages that describe what the platform does, who it’s for, and what it costs.

If those pages aren’t in the training data or retrieval index, you’re not on the shortlist.

Why BOFU pages are structurally under-crawled

Here’s the nuance worth understanding: BOFU under-coverage isn’t always a sign that something is broken. It’s frequently a direct consequence of how current AI crawlers are built.

GPTBot’s breadth-first crawl pattern prioritizes discovery breadth over depth. It finds the most-linked pages on your site and spreads from there. On most business sites, /blog/ and /about/ have the most inbound internal links. /pricing/ typically has fewer. The crawler follows the link graph, not your conversion funnel logic.

The BOFU gap is, in many cases, an artifact of normal internal link architecture rather than a malfunction.

Fixing it requires both content-layer work (building BOFU pages that earn more internal links and contain more retrievable content) and structural work (ensuring AI bots can access BOFU sections cleanly without JavaScript render requirements, rate-limit blocks, or robots.txt edge cases). Read the agentic search optimization guide for the full framework for what to do once you’ve identified the gaps.

Shadow crawl: the BOFU demand signal you’re probably not reading

Shadow crawlers, ChatGPT-User and Perplexity-User specifically, are on-demand fetchers that fire when a real user asks an AI assistant a question referencing your site. They operate outside the scheduled crawl cycle and aren’t subject to robots.txt, because the user, not the bot, is the entity making the request.

That same study measured a median of approximately 690 combined hits per site per day from ChatGPT-User and Perplexity-User across twelve production sites. Those aren’t training signals. Those are real users asking an AI something about your site, right now.

Where those hits land in your funnel matrix is one of the most valuable pieces of data in your logs.

• Shadow hits on /blog/ mean people are asking AI assistants about your content topics.
• Shadow hits on /pricing/ or /demo/ mean people are actively evaluating you as a vendor through an AI interface.

A site with heavy GPTBot TOFU coverage but zero shadow-crawl BOFU has an awareness problem. A site with shadow crawl activity on /pricing/ already has an agentic audience at the decision stage, regardless of whether the training crawlers have caught up.

Segment shadow crawl hits separately in your funnel matrix. They tell a fundamentally different story than the scheduled crawlers.

For context on how shadow crawl relates to actual AI-referred traffic, the getting traffic from AI guide covers the referral attribution side of the same picture.

Funnel gap diagnostic table

From funnel gaps to agentic optimization: four actions

Four actions turn a funnel gap analysis into a concrete optimization plan.

Funnel analysis produces a diagnostic map. Here’s how to act on it.

Action 1: Fix access before fixing content

If your funnel matrix shows MOFU and BOFU near-zero, verify access before you touch a single word of content.

• Check your robots.txt for path-level restrictions on /features/, /pricing/, or /solutions/. These exist more often than they should, usually as legacy configuration artifacts nobody cleaned up.
• Check your CDN or WAF for bot-fight mode settings that may be blocking AI user agents at the network layer.
  • Cloudflare’s bot fight mode settings can silently block AI user-agents depending on your configuration. If your site uses Cloudflare, you should explicitly verify which crawlers are allowed or blocked.

Action 2: Increase internal link signals to MOFU and BOFU sections

GPTBot follows the link graph. If your /pricing/ page has three internal links pointing to it and your top blog posts each have fifty, GPTBot’s visit distribution will reflect that ratio. Audit internal link counts by section and systematically add contextually relevant links from high-hit TOFU pages to MOFU and BOFU destinations.

This isn’t gaming the crawler, it’s ensuring the crawler has a path to the content that actually matters for vendor evaluation.

Action 3: Improve content retrievability and agent executability on BOFU pages

BOFU pages that rely on client-side JavaScript for their primary content are invisible to most AI crawlers, which don’t execute JavaScript. Pricing tables built in React components, feature comparisons loaded via API calls, and demo CTAs injected by marketing automation are all at risk.

Server-render or pre-render any content on BOFU pages that you want AI systems to access. Run those pages through a raw HTTP request (curl or wget) and verify that the critical content actually appears in the response body, not just in a rendered browser view. If it doesn’t show up in curl, AI bots can’t see it either.

The same principle applies to the action layer, but the failure mode is different.

A page can be fully server-rendered and perfectly readable by a training crawler while still being completely unusable by an AI agent trying to execute a task. A demo booking flow that requires a multi-step JavaScript modal, a trial signup that depends on a third-party form widget, or a purchase checkout that doesn’t expose machine-readable form fields.

All of these are action layer failures that won’t show up as crawl gaps in your logs. They show up as completed visits with no downstream conversion, and you’d need to test them manually with an agent to find them. The log data tells you that an agent reached the page. Only a task-completion test tells you whether it could do anything once it got there.

Action 4: Treat shadow crawl on BOFU as a conversion signal

If your funnel matrix shows ChatGPT-User or Perplexity-User hitting your /pricing/ or /demo/ pages, you already have agentic buyers at the decision stage. The question isn’t how to get them there, since they’re already there. There are now two questions to answer, not one.

First, the knowledge question:

if someone asked an AI to evaluate your product and the AI fetched your pricing page, what answer would it synthesize from what it found? Does the page give clear pricing context, specific use-case fit signals, and a structured call to action? Run that test manually. The answer tells you more than most CRO audits.

Second, the action question:

• If someone instructed an AI agent to book a demo or start a trial on your site, would it succeed?
• Can the agent identify and interact with the booking form?
• Does your demo request flow require JavaScript that the agent can’t execute?
• Is there a machine-readable confirmation state that the agent can report back to the user?

These are testable questions today. Open a capable AI agent, give it the instruction, watch what happens, and note exactly where it fails. That failure point is your action layer gap.

Use the Similarweb Site Audit tool to track and analyze server log files, and Similarweb’s AI Search Intelligence to track whether BOFU page improvements translate into increased citation share on vendor-evaluation queries.

This is the measurement layer that connects the log diagnostic to actual business outcomes.

Action layer audit checklist

Before calling your funnel analysis complete, run these five manual tests with a capable AI agent:

1. Can the agent locate and interact with the demo booking form on /demo/?
2. Does the trial signup flow complete without JavaScript modal dependencies?
3. Does the pricing page render its primary content in a raw curl request?
4. Is there a machine-readable confirmation state for completed bookings?
5. Can the agent identify your product category and primary use case from the /features/ page in one pass?

Running a funnel analysis is not optional anymore

Twenty years of SEO have conditioned us to optimize for Googlebot because Google controls organic discovery. That logic doesn’t disappear in agentic search. it compounds.

Googlebot still matters. But now, four additional crawler families also matter for AI-mediated discovery, each with different path preferences, crawl shapes, and implications for which stage of your funnel AI systems understand.

The funnel matrix is a one-hour analysis that tells you whether AI systems know enough about your product to recommend you. Most sites I’ve looked at have strong TOFU coverage and significant MOFU and BOFU gaps, not because anything is broken, but because the content distribution that works for traditional SEO hasn’t been extended to the commercial architecture that serves agentic recommendations.

The action layer is the next frontier. Knowing that an agent reached your pricing page is the starting point. Knowing whether it can complete a demo booking once it gets there is the question that separates sites optimized for agentic search from those merely crawled by it.

Once you have the map, the optimization is straightforward. The GEO audit framework covers the broader content and technical audit that sits alongside this log diagnostic. The data is already in your logs. Most teams just haven’t asked the right question yet.

FAQ

What is AI bot funnel analysis?

AI bot funnel analysis is the practice of segmenting server log data by site section and reading AI bot crawl patterns as a buyer journey funnel to identify where AI systems’ knowledge of your site ends. It operates across two layers. The knowledge layer asks whether AI systems have crawled your site thoroughly enough to recommend you as a vendor: distinguishing coverage at the awareness stage (TOFU) from coverage at the evaluation and decision stage (BOFU). The action layer asks whether AI agents can actually execute tasks on your site, such as booking a demo or completing a purchase, once they arrive. Both gaps show up in your logs, but they require different diagnostics and different fixes.

What is the difference between the AI knowledge layer and the AI action layer?

The knowledge layer is about what AI systems have learned about your site through crawling: do they know what your product does, who it serves, and what it costs? A gap here means an AI won’t recommend you. The action layer is about whether AI agents can complete tasks on your site on a user’s behalf: can they navigate a demo booking form, initiate a trial, or step through a purchase flow? A gap here means an AI might recommend you but the agent fails at the conversion step. Logs tell you which pages agents are reaching. Only manual task-completion testing tells you whether they can do anything useful once they get there.

Why do AI bots crawl blog pages more than product and pricing pages?

AI training crawlers like GPTBot operate breadth-first and prioritize the most-linked pages on a site. On most business websites, /blog/ and /about/ sections have the highest internal link counts, so they receive the highest bot visit frequencies. /pricing/ and /demo/ pages consistently receive materially lower AI bot hit rates, a direct consequence of how breadth-first crawlers follow the internal link graph, not your conversion funnel logic. Correcting it requires improving internal link paths to BOFU sections and ensuring those pages are server-rendered and technically accessible.

What does shadow crawl data tell me about my agentic search visibility?

Shadow crawlers, specifically ChatGPT-User and Perplexity-User, are on-demand fetchers that fire when a real user asks an AI assistant a question referencing your site. Unlike scheduled training crawlers, they represent active demand: someone is querying an AI about your domain right now. Segmenting shadow crawl hits by site section tells you which funnel stages are being evaluated by real users through AI agents. Shadow crawl activity on /pricing/ or /demo/ pages indicates agentic buyers at the decision stage, regardless of what scheduled crawlers are doing.

How do I identify funnel gaps in my AI bot log data?

Filter your server logs for AI bot user-agents, extract the top-level URL path prefix from each hit, and count hits by section and bot type. The result is a matrix in which low or zero values in the MOFU (/features/, /solutions/) or BOFU (/pricing/, /comparison/) columns indicate funnel gaps. Cross-reference your sitemap to confirm those pages exist and aren’t blocked by robots.txt, then check for technical access issues, CDN bot-fight mode, JavaScript rendering requirements, and rate-limiting before moving on to content optimization.

Can I improve my BOFU AI bot coverage through content alone?

No, not if the pages have technical access barriers. Verify first: CDN and WAF settings aren’t blocking AI user-agents on those URL paths, robots.txt doesn’t exclude BOFU paths, and page content is present in the server response body rather than loaded client-side via JavaScript. Once access is confirmed, BOFU AI coverage improves through content depth (clear pricing context, use-case specificity, structured comparison content), schema markup, and internal link signals from high-hit TOFU pages pointing to BOFU sections.

How often should I audit AI bot behavior in server logs?

Monthly for most sites. Immediately after major content changes, product launches, or site infrastructure updates. AI bot crawl behavior shifts with model updates, and a monthly funnel matrix review establishes the baseline and surfaces behavioral changes before they become invisible GEO problems. Sites running active agentic search optimization programs benefit from weekly review cycles for the first 90 days.

What is the difference between OAI-SearchBot and GPTBot in server logs?

GPTBot is OpenAI’s training crawler, collecting content to improve foundational AI models. OAI-SearchBot is OpenAI’s search crawler, fetching pages when ChatGPT performs a web search query. After the GPT-5 launch in August 2025, OAI-SearchBot activity grew roughly 3.5x and now generates more log events than GPTBot, a reversal of the prior pattern, per the Botify/Chris Long analysis of approximately 7 billion OpenAI log events (November 2024 to March 2026). For funnel analysis, OAI-SearchBot hits on your BOFU pages are a stronger GEO signal than GPTBot hits because they reflect active search retrieval rather than passive training collection.

What about Bytespider and Applebot?

As of mid-2026, Bytespider (ByteDance) and Applebot have both broken into the top five AI crawlers by traffic share, per Cloudflare Radar data. Their GEO implications differ materially from the crawlers covered in this article. Bytespider has no referral mechanism, meaning it collects content for Meta AI features without sending any traffic back to publishers. Applebot serves Apple Intelligence and Siri, with GEO implications that are still emerging as Apple’s AI product surface grows. Both warrant monitoring in your funnel matrix, but their optimization implications are distinct enough to require separate treatment beyond this article’s scope.