An anonymized marketplace analysis found 82% recrawls, 39% parameter waste, and heavy duplicate-mobile leakage. Here is how to turn those signals into fixes.
High crawl volume did not mean the site was being explored well. In this case, the better signal was where crawler effort actually landed: URLs that were already known, duplicated, parameterized, paginated, or otherwise unlikely to improve discovery of canonical, indexable, commercially important pages. That is also the practical reading of Google’s guidance on crawl budget: not “more hits is better,” but whether crawling is spent on URLs worth fetching at scale (official documentation, managing crawl budget for large sites).
The crawler was busy, but much of the work had little value
Within the sampled snapshots, about 82% of crawler activity revisited known URLs, more than 39% reached parameterized URLs, duplicate mobile URLs consumed more than 50% of product-detail crawl activity, and pagination absorbed about 40% of category and listing crawl activity. Taken together, the crawler was active, but much of that activity went to low-yield paths.
Waste signal from sampled snapshots
What it means operationally
Why it lowers crawl value
82% of crawler activity revisited known URLs
Most requests returned to URLs already discovered
Discovery capacity for new or corrected canonicals was limited
More than 39% of requests hit parameterized URLs
Filters, sorts, tracking, or session-like variants were being fetched
Crawl was diluted across alternate URL states
More than 50% of product-detail crawl hit duplicate mobile URLs
Product attention split across duplicate representations
Canonical product URLs received less concentrated crawl
About 40% of category/listing crawl went to pagination
Deep listing sequences absorbed bot time
Higher-value category hubs and key products competed for fetches
The key question is simple: what share of crawl was clearly aligned with the pages that matter most? If about 82% was repeat work, only about 18% was not repeat work in that sample. And inside major templates, that remainder was still diluted. More than 50% of product-detail crawling went to duplicate mobile URLs, while about 40% of category and listing crawling went to pagination. So before anyone celebrates crawl growth, they need to check where those requests landed and whether those URLs were canonical, indexable, and commercially meaningful.
Where crawl budget goes to waste — Convertos.ai original workflow poster.Where crawl budget goes to waste — Convertos.ai original explainer.
This original 34-second narrated explainer reduces the article to three checks: Inventory URLs, Verify in logs, Tighten discovery. It summarizes the article and does not carry essential claims that are absent from the text. Captions and a transcript are included for accessibility.
Video transcriptEcommerce sites waste crawl budget when bots spend time on filters, sorting pages, internal search, tracking parameters, and duplicate paths. The problem is not simply having many URLs. It is making low-value URLs easier to discover than products and categories that matter. Inventory the URL patterns first. Use server logs to confirm what Googlebot actually fetches. Then tighten internal links, parameter handling, canonicals, and sitemap rules. Measure the share of crawl activity reaching valuable pages again after two to four weeks.These shares have different denominators, but each identifies a large source of low-value crawl.
Four URL patterns consumed the crawl opportunity
The waste was not random. In the sampled snapshots, four URL patterns repeatedly pulled crawler time away from canonical, indexable pages that could actually rank: known-URL recrawls, parameterized URLs, duplicate mobile paths, and pagination. These figures come from limited observation windows, so they show concentration of activity, not controlled causation. They are still useful because each pattern points to a different generator and a different fix.
First, the crawler kept returning to URLs it already knew. About 82% of crawler activity in the sample revisited known URLs instead of discovering net-new ones. That is only healthy when the revisits land on pages worth refreshing. If they land on duplicates, thin variants, or dead listings, crawl volume rises while useful coverage stalls. Google’s guidance for managing crawl budget for large sites makes the same point: the issue is not raw bot traffic, but whether crawling is focused on URLs that matter.
Parameters were the clearest multiplier. More than 39% of crawler hits reached parameterized URLs. That suggests sort, filter, tracking, or session-like variants were being emitted into crawlable paths faster than canonical targets could absorb demand. On large catalogs, this is how one product or category becomes dozens of fetchable addresses. Google’s documentation on faceted navigation URLs exists for exactly this problem.
Duplicate mobile paths created a second copy of product-detail demand. More than 50% of product-detail crawl activity went to duplicate mobile URLs rather than the preferred product URLs. That is not a mobile strategy. It is a crawl split. If half of product-detail fetching goes to alternates, the canonical set gets only the remainder before any other duplication is counted.
Pagination did the same thing to category and listing pages. About 40% of category and listing crawl activity was absorbed by paginated URLs. In a later snapshot, non-200 category/listing URLs consumed 14% of crawl budget, up from about 9%, while the number of 4xx URLs found in the site structure grew 3.5 times. That combination is a warning sign because it shows the crawler spending more time on broken or low-value paths.
Pattern
Share seen in sample
What it displaced
Known-URL recrawls
About 82% of crawler activity
Discovery and refresh of priority canonicals
Parameterized URLs
More than 39% of crawler activity
Clean category and product URLs
Duplicate mobile product paths
More than 50% of product-detail crawl
Preferred product-detail canonicals
Pagination
About 40% of category/listing crawl
Core listing pages and valid deeper inventory
A rising crawl count can be a warning, not a win
A higher crawl count can look like progress in a dashboard. It can also mean URL hygiene is getting worse.
In the later snapshot here, non-200 category and listing URLs took 14% of crawl activity, up from about 9% in the earlier snapshot. That is a 5 percentage point increase in crawler effort spent on pages that did not return a successful page load. Over the same comparison window, discovered 4xx URLs in the site structure increased 3.5x. These are snapshot comparisons, not a controlled experiment, so they show association rather than proof of cause. But they are strong warning signals because they show where the crawler actually went.
The issue is not that crawlers ever touch error URLs. Some non-200 activity is normal on large sites. The problem starts when that share rises while commercially useful coverage does not. If a crawler is spending more requests on broken category paths, expired listings, or malformed variants, then a bigger total crawl number can hide a smaller share reaching canonical, indexable pages. That is why Google frames crawl budget as a question of whether Googlebot can spend time on the URLs that matter, not whether raw request volume is high in isolation. See Google’s official documentation and Crawl Stats.
Snapshot
Non-200 share on category/listing crawl
Change vs earlier
Discovered 4xx URLs in site structure
Earlier sample
About 9%
Baseline
Baseline
Later sample
14%
+5 percentage points
3.5x
A practical rule helps: treat rising crawl volume as a risk when error discovery grows faster than useful URL coverage. On a large site, that can delay recrawl of pages that actually drive revenue.
Start with logs, then prove the URL rule
If crawl waste is suspected, start with raw request logs, not aggregate crawl charts. The job is to classify each bot hit by URL pattern and page intent, then test whether a generator rule is creating low-value demand. Pull a representative log sample, isolate verified Googlebot requests, and join each requested URL to six fields: status code, canonical target, indexability, page type, template or route, and business value tier. Google’s guidance on managing crawl budget for large sites is useful here because it treats crawl budget as a routing problem, not a vanity metric.
A practical schema looks like this:
Field
Example value
Why it matters
Bot
Googlebot Smartphone
Separates search crawl from other automation
URL rule
?sort=, /m/, ?page=
Lets you test generator-level causes
Status
200, 301, 404
Quantifies wasted fetches and broken paths
Canonical state
self, other URL, none
Shows whether crawl lands on the preferred URL
Indexability
indexable, noindex, blocked, duplicate
Distinguishes useful discovery from dead ends
Page type
product, category, pagination, search
Connects waste to templates
Value tier
high, medium, low
Prevents equal treatment of unequal pages
Once that join exists, compare crawl share against page value. In one later snapshot from this case, more than 39% of crawler activity reached parameterized URLs, duplicate mobile URLs consumed more than 50% of product-detail crawl activity, and pagination absorbed about 40% of category and listing crawl activity. Those figures come from snapshots, not a controlled experiment, so they show concentration, not causation. But they are enough to test a generator hypothesis: if a route pattern repeatedly attracts Googlebot while resolving to duplicate, non-canonical, or low-value states, the template is manufacturing crawl demand.
Use a simple decision rule: prioritize a URL rule when it takes a visible share of Googlebot requests and those requests often end on duplicate, non-canonical, blocked, or otherwise low-value pages. For faceted and parameterized paths, Google’s documentation on faceted navigation URLs gives the implementation guardrails, but the proof should come from your own logs. Logs show what Googlebot actually requested. The URL rule shows what the site kept generating.
Fix the generator before cleaning individual URLs
When repeat crawl dominates, deleting URLs one by one is usually the slowest fix. The better move is to stop producing low-value paths in the first place. In this case, the pattern-level waste was clear within the sampled snapshots: more than 39% of crawler activity hit parameterized URLs, duplicate mobile URLs took more than 50% of product-detail crawl activity, and pagination absorbed about 40% of category and listing crawl activity. Those shares do not prove causation across all periods, but they are enough to set engineering priority. If a template, link module, or routing rule keeps generating these paths, manual cleanup will be overwritten by the next deploy.
Start where the site creates discoverability. Internal links matter more than isolated URL removals because they teach crawlers what exists and what deserves revisits. If filters, sort orders, session-like parameters, or alternate mobile paths are linked from navigation, cards, breadcrumbs, or faceted controls, the crawler will keep finding them. Google’s guidance on faceted navigation URLs and consolidate duplicate URLs points in the same direction: reduce crawlable combinations, consolidate duplicates, and make the preferred URL the easiest one to discover.
Rank generators by recoverable crawl share, not by how messy the URLs look.
Generator to fix
Evidence from sampled snapshots
Why it goes first
Typical rule change
Mobile duplicate product URLs
More than 50% of product-detail crawl
High-value template, direct collision with canonical product crawl
Remove crawlable alternate mobile paths; unify internal links to one canonical product URL
Parameterized URLs
More than 39% of total crawler activity
Sitewide waste across many templates
Block unnecessary parameter generation in links and forms; keep only required crawlable states
Pagination
About 40% of category/listing crawl
Large recurring drain in listing templates
Limit deep crawlable pagination and reduce linked page combinations
Error-producing category/listing templates
Non-200 share rose from about 9% to 14%; 4xx URLs in structure grew 3.5x
Waste is increasing, so the generator is still active
Fix broken link logic, retired nodes, and error templates exposing dead paths
Sitemaps
Fast way to stop re-advertising low-value URLs
Include only canonical, indexable 200 URLs
The decision rule is straightforward: fix generators that control a meaningful share of crawl in commercially important templates before touching individual URLs. The later snapshot makes the case stronger because non-200 category and listing URLs rose from about 9% to 14%, while 4xx URLs exposed in the site structure grew 3.5 times. That is not just cleanup debt. It points to a production source still emitting bad paths.
Measure where crawl moved after the release
The release is not validated by a lower crawl count. It is validated when crawl shifts toward canonical, indexable, commercially important URLs. Keep the scorecard small enough to review weekly from logs, Search Console, and sitemap exports. Four measures usually tell the story: useful-crawl share, discovery latency for new priority URLs, indexable coverage of priority templates, and ranked or clicked outcomes on the same URL set. Google’s guidance on managing crawl budget for large sites supports that framing: the goal is not more fetching, but better fetching.
Metric
How to calculate
Why it matters
Recording field
Useful-crawl share
Googlebot hits to canonical, indexable priority URLs ÷ all Googlebot hits
Shows whether crawl moved away from waste
Pre-release: ___ / Post-release: ___
Discovery latency
Time from first internal link or sitemap inclusion to first Googlebot hit on new priority URLs
Tests whether important new pages are found faster
Pre-release: ___ / Post-release: ___
Indexable coverage
Priority URLs returning 200, self-canonical, not blocked, and eligible for indexation ÷ all priority URLs
Separates crawl improvement from template or status-code defects
Pre-release: ___ / Post-release: ___
Search outcome rate
Priority URLs with impressions, rankings, or clicks ÷ crawled priority URLs
Checks whether better crawl reached pages that can perform
Pre-release: ___ / Post-release: ___
The worked example is the one that matters. In the observed snapshots, about 82% of crawler activity revisited known URLs, more than 39% reached parameterized URLs, duplicate mobile URLs took more than 50% of product-detail crawl activity, and pagination absorbed about 40% of category crawl activity. Those figures come from limited snapshots, not a controlled test, so treat them as directional. If a release reduces those waste patterns, useful-crawl share should rise even if total hits stay flat.
Do not stop at logs. Pair the shift with coverage and outcome checks. If non-200 category and listing URLs still consume 14% of crawl in a later snapshot, up from about 9%, or if 4xx URLs in the site structure have grown 3.5 times between snapshots, the generator is still leaking crawl opportunity. Use Crawl Stats to confirm host-level changes, but judge success on the priority URL set. The concrete decision is this: if useful-crawl share does not rise while non-200 share and discovered 4xx exposure stay elevated, keep work focused on generator-level fixes rather than individual URL cleanup.
Keep the retest on the same cohort in the SEO audit workflow; the Convertos SEO guides can then guide the page-level follow-up without changing the baseline.
Disclosure
The case data comes from a private 2026 operating review of a large cross-border marketplace. It is reproduced with permission after company, vendor, domain, system, and personnel identifiers were removed. The figures show association, not controlled causation.
