How to Use AI for Log File Analysis and Crawl Budget on Large Stores

To use AI for SEO log file analysis on an ecommerce site with a crawl budget problem, split the work in two: code verifies and counts Googlebot's requests, and the model takes the job that never gets finished by hand, which is naming every URL pattern in the log. A store that has been through a couple of replatforms carries years of filter, sort and tracking parameters, and nobody remembers what half of them do. A language model is well suited to reading sample URLs and labeling them, while the counting stays with code.

The workflow runs in four passes: verify, classify, measure against Google's own definitions, then fix and check. A hypothetical home goods store runs through it, and every number attached to that store is an assumption. Crawl access sits upstream of every visibility measure we write about, AI share of voice included, because a product page has to be fetched before any system can consider it.

What Google means by crawl capacity and crawl demand

The crawl budget guide for large sites that SEOs know from Google Search Central rests on two terms, crawl capacity limit and crawl demand, and faceted navigation is where log files most often show the second one going wrong. The guide now sits in Google's crawling infrastructure documentation; the old Search Central address redirected there when we checked on October 7, 2026.

Google for Developers says the guide is intended primarily for large sites (1 million+ unique pages) with content that changes about once a week, medium or larger sites (10,000+ unique pages) with content that changes daily, and sites with a large portion of URLs that Search Console classifies as "Discovered - currently not indexed". It calls those numbers a rough estimate.

The crawl capacity limit, per the same page, limits the total amount of time a server spends holding connections open for Google. It goes up when the site responds consistently and goes down when responses slow or the server returns 5xx errors or HTTP 429. Crawl demand for Googlebot varies with a site's size, update frequency, page quality and relevance, and the primary factors a site owner can influence are perceived inventory, popularity and staleness. Of perceived inventory the guide says: "This is the factor that you can positively control the most." Taken together, Google defines a site's crawl budget as "the set of URLs that Google can and wants to crawl." The log has to answer both halves: whether the server is holding capacity down, and how much demand is pointed at URLs the store never wanted crawled.

Start with requests that came from Google

Any client can claim to be Googlebot in its user agent string, so the first pass filters the log down to requests that came from Google's own network. Google for Developers documents two methods. One-off checks use a reverse DNS lookup on the logged IP address, which should return a googlebot.com, google.com or googleusercontent.com name, then a forward DNS lookup on that name, which should return the original IP address. Large scale lookups match each IP address against Google's published lists of crawler IP ranges.

A month of logs from a large store calls for the second method, in a script with no model involved. Keep the crawler type as a column. The crawl budget guide says the crawl capacity limit is shared across all crawlers and that Google Shopping has a higher demand for products in merchant feeds, so a store with a product feed should size each crawler's share of the load before attributing all of it to Googlebot. The same log, filtered on other user agents, is where the AI crawler check in our guide to getting cited by ChatGPT begins.

Where the model earns its place

AI-based log file analysis of an ecommerce crawl budget works best when the model never sees the full log. Collapse the verified requests into distinct patterns first: replace IDs and slugs in the path with a placeholder, drop the parameter values, keep the parameter keys in sorted order, and attach a request count and three sample URLs to each pattern, with any emails, names or tokens in the samples masked. For the hypothetical store, assume 4,200,000 verified requests over 30 days collapse into a few thousand patterns, a list short enough to send in batches.

We would start from a prompt like this:

You are labeling URL patterns from an ecommerce server log.
Each row has: pattern_id, path template, sorted parameter keys,
three sample URLs, and a count of verified Googlebot requests.

Return one JSON object per row with these fields:
- pattern_id
- template: product | category | category_paginated | facet_single |
  facet_multi | sort_or_view | internal_search | cart_or_account |
  static_resource | other
- parameter_roles: each key mapped to filter | sort | pagination |
  tracking | session | search | unknown
- confidence: high | medium | low
- reason: one short sentence

Label from the pattern and the samples only. If the samples do not
settle a parameter's role, answer unknown. Do not decide whether a
URL should be crawled or indexed.

Two lines carry most of the weight. The "unknown" option gives the model a way out, and every unknown or low-confidence row goes to a person who can load the URL. The closing instruction keeps the crawl decision with the SEO team, since whether a color filter deserves crawling depends on search demand that a log cannot show.

Ask for schema-constrained output where the model's API offers it. Anthropic's Claude Platform Docs, for one, say structured outputs constrain Claude's responses to follow a specific schema, and the same page lists cases where the output may still miss the schema, including a refusal or a response cut off at the max_tokens limit. Count the rows that come back and validate each one before joining the labels to the full log.

Reading wasted crawl from the classification table

AI log file analysis starts paying off for SEO and crawl budget once the labels are joined back to the request counts. Every figure in the table below is an assumption for the hypothetical store.

The model supplies the class labels, and code supplies the patterns and the counts; the redirects and errors row comes straight from the log's status codes. The three Block rows add up to 47% of verified requests, or 1,974,000 of 4,200,000, spent on URLs the store does not want crawled. In Google's terms, that figure is perceived inventory with a number on it.

The capacity side comes from the same file. Chart the daily response time for verified requests and the daily share of 5xx and 429 responses, the signals Google says pull the capacity limit down. Flat lines point to a demand problem alone, while spikes on the days facet crawling peaks mean the waste is costing capacity as well.

What Google documents for faceted navigation and URL parameters

For ecommerce sites, Google Search Central's guidance on faceted navigation, URL parameters and crawling comes in two branches, chosen by whether the filtered pages need to appear in search. In December 2024 the Google Search Central Blog wrote that "faceted navigation is by far the most common source of overcrawl issues site owners report to us".

When the filtered URLs do not need to be indexed, Google for Developers recommends preventing the crawl, either with robots.txt disallow rules on the filter parameters or by moving the filters into URL fragments. The page describes rel="canonical" and rel="nofollow" as generally less effective in the long term than those two methods. When the filtered URLs do need to be crawled, it asks for three things: use & as the parameter separator, keep the logical order of filters the same if they are encoded in the URL path, and return an HTTP 404 status code when a filter combination returns no results.

Search Console's URL Parameters tool is no longer an alternative. The Google Search Central Blog announced its deprecation on March 28, 2022, reported that only about 1% of the parameter configurations then specified in the tool were useful for crawling, and pointed site owners who need more control to robots.txt rules.

Fixing what the log exposed

Fixing crawl budget problems on a large ecommerce site, once log file analysis has sized them, means working down the Decision column in order of request volume. For the Block rows the model drafts and code tests:

From the approved Block rows below, draft robots.txt rules for
user-agent: Googlebot using only the * and $ wildcards.
For each rule, list three logged URLs it must match and three
Keep-class URLs it must leave crawlable.
Flag any rule that could also match a Keep or Review pattern.

For the hypothetical store, a draft could block every URL carrying a sort, view, price or size parameter and leave color-only URLs crawlable while the Review row is decided:

user-agent: Googlebot
disallow: /*?*sort=
disallow: /*?*view=
disallow: /*?*price=
disallow: /*?*size=
disallow: /search?

Treat the model's match claims as untested until a robots.txt parser has run every rule against every pattern in the table. Google for Developers specifies that * stands for 0 or more instances of any valid character, that $ marks the end of the URL, and that rule values are case-sensitive. Read literally, /*?*size= also matches a pagesize= parameter, and a bare /search rule would match every path that starts with those characters. Collisions like these are what the test list is for. The same page says only one group is valid for a particular crawler, so rules addressed to Googlebot do not bind Storebot-Google, the shopping crawler, which needs its own group if the log shows it on these patterns.

The rest of the column follows the two Google pages already cited. The crawl budget guide advises against noindex for this job, because Google still requests the page before dropping it, and calls for a 404 or 410 on permanently removed pages. The faceted navigation page wants a 404 on empty filter combinations, served under the URL that was requested.

Two cautions belong in the plan. Blocked URLs, the guide says, stay part of the crawl queue much longer and are recrawled when the block is removed. On the payoff it is direct: "Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit." The dependable gains are fewer wasted requests and a lighter server load, with faster product discovery most likely where capacity was the constraint. Engineering hours for this tend to come out of the same pool as answer-engine work, a trade our piece on splitting a budget between SEO and AI answer visibility covers.

The before and after check in Crawl Stats

In Search Console the report sits under Property settings, then Crawl stats. Search Console Help says it is available only for root-level properties and that it counts the actual URLs Google requested, without assigning data to canonical URLs. Before the fix ships, record total crawl requests, average response time, and the breakdowns by crawl response, crawl purpose (Discovery or Refresh) and Googlebot type.

Google generally caches robots.txt for up to 24 hours, according to the robots.txt specification page cited above, so the first day after the change says little. Search Console Help adds that a crawl rate should be relatively stable over a week or two, which makes matching two-week windows a sensible comparison. It also lists a known issue under which the report can differ slightly from a site's request logs, so the log stays the measurement and Crawl Stats is the cross-check. If the Block share is still well above zero after two weeks, check first whether a rule failed to match.

For further resources, see our SEO checker and AI visibility checker, or talk to the AI search optimization team about running this audit alongside the rest of a search program.

Sources

• Google for Developers: Optimize your crawl budget, last updated July 22, 2026
• Google for Developers: Managing crawling of faceted navigation URLs, last updated December 18, 2025
• Google for Developers: Verify requests from Google crawlers and fetchers, last updated March 20, 2026
• Google for Developers: How Google interprets the robots.txt specification, last updated August 31, 2026
• Google Search Central Blog: Crawling December: Faceted navigation, December 17, 2024
• Google Search Central Blog: Spring cleaning: the URL Parameters tool, March 28, 2022
• Search Console Help: Crawl Stats report
• Claude Platform Docs: Structured outputs