While reviewing crawl data for an ecommerce site, I noticed a group of 43.1 million requests. They went to addresses classified as "clean URLs," yet those addresses didn't appear in the inventory produced by a site crawl.
That group deserves a closer look. Discussions of wasted crawling tend to focus on parameter URLs. But a substantial group of clean URLs also sat outside this inventory. Checking whether an address has parameters won't explain that gap.
During the same period, clean URLs inside the inventory received just 4.7 million requests, or 3.2% of all Googlebot requests. That looks low. Before deciding what to change, though, I'd want to know what the pages outside the inventory actually were.
I'd investigate those 43.1 million requests before calling them waste
The two clean-URL groups received 47.8 million requests in total. About 90.2% of those requests went outside the inventory. Even with parameter URLs set aside, most clean-URL requests still don't match the crawl inventory.
That doesn't tell me nine out of ten clean pages were missed by the site crawler. We're counting requests here. A small number of addresses visited repeatedly could account for a large share of them. To measure the number of pages involved, I'd need distinct URL counts.
"Clean" is also only a category used here to distinguish these addresses from parameter URLs. An address without parameters can still lead to a discontinued product, duplicate content or a redirect. Whether it deserves attention depends on the page itself.
The 3.2% figure therefore tells me what share of requests matched the inventory. Calling it the share of useful crawling would require another judgment: which pages were worth crawling? A large group of live, useful product pages outside the inventory would call for a very different response from a large group of obsolete addresses.
I'd check what each dataset includes
The logs cover 30 days. The site crawl is dated September 7. They don't capture the same activity over the same period.
Suppose a product was removed in late August. Googlebot may have visited it before removal, while the September 7 site crawl could no longer find a link to it. That address could end up outside the inventory. A page first published on September 8 would also be absent from the previous day's crawl. Publication and removal records would help test these possibilities; the chart alone can't distinguish them.
I'd also check where the crawl started, whether it excluded directories or language versions, whether it reached a URL limit, and whether it completed. Those settings affect which pages it can discover. An inventory of 2 million pages is large, but its size doesn't establish that it covers the whole site.
Then I'd inspect the URL matching. One export may contain paths while another includes the hostname. One may record the original request address while another records the destination after a redirect. I'd make sure both sides refer to the same thing. Stripping parameters, language paths or trailing slashes just to improve the match rate could merge different pages, so I'd keep the original addresses and check which forms are actually equivalent.
Google's knowledge of a site also extends beyond a single site crawl. Its explanation of URL discovery describes addresses known from previous visits, links from other pages and submitted sitemaps. An address that can't be reached through the site's current links may still be one Google already knows.
The pages I find would determine the work
I'd start by grouping those 43.1 million requests by URL to see where the volume concentrates. Alongside the most frequently requested addresses, I'd sample different directories, language versions and page types so the investigation doesn't cover only a handful of busy URLs.
For each address, I'd check what it returns now, whether its content is still useful, and how a visitor could reach it. The product catalog and sitemaps can contribute candidate URLs, but I'd still need to inspect the pages and their internal links. A sitemap entry doesn't show that visitors can easily find a page on the website.
Different findings would lead to different work.
| What the investigation finds | What I'd do next |
|---|---|
| The page is useful and reachable through normal internal links, but absent from the crawl inventory. | Check crawl scope, filters and URL matching. Correct the inventory before deciding whether the site needs a change. |
| The page is useful, and further checks confirm that it lacks a relevant internal entry point. | Add a link from an appropriate category, collection or related product, then verify that the page can be discovered through that route. |
| The address leads to discontinued content, an old page or a redirect. | Decide whether to retain, redirect or remove it based on the available replacement content, and check the current response. Don't restore links to every old address just to get it into the inventory. |
This is why I wouldn't label the whole group "orphan pages." I'd need to investigate internal links before concluding that entry points were missing. If a crawl excluded a directory, adding links wouldn't address the cause of the mismatch.
Sampling can point to the next investigation, but I wouldn't multiply a sample percentage by 43.1 million to estimate wasted requests across the site. To explain the August and September requests, I'd need product status and response information from when those requests occurred. A product that is no longer available today may still have been useful then. Only after checking those timelines would I apply classification rules to the full logs and count the distinct URLs and requests involved. Unresolved addresses would remain unclassified.
A better match rate can come from a better inventory
Suppose another site crawl finds pages missing from the first inventory. Match that expanded list against the same historical logs, and the share of requests inside the inventory can rise. Googlebot doesn't have to make a single new request for the percentage to change.
I therefore wouldn't make "raise 3.2% to a target percentage" the repair goal. A fuller inventory gives me a better account of where the requests went. Whether the website improved is a separate question.
If the investigation found important products without internal entry points, I'd save the list of pages to fix before making changes, along with their publication or content-update dates. Afterward, I'd track that same group over observation windows of equal length, using the same log source: how many pages received a successful content fetch, and how long after publication or an update did it happen? Pages still uncrawled at the end would stay in the results as "still waiting." I'd avoid calculating waiting times only for pages that had already been crawled.
New products could be tracked separately rather than continually added to the original list. Keeping the group fixed reduces one source of confusion, but a before-and-after difference alone won't establish that adding links helped. I'd also check page updates and changes in crawling across the site. If crawling became faster everywhere, I couldn't attribute this group's improvement entirely to the new links.
Whether those pages were then indexed or received search clicks would need its own follow-up. Request logs can't answer those questions.
I'd begin with a sample of the clean URLs outside the inventory and work out whether the gap comes from missing inventory entries, missing internal routes or old addresses still being requested. That would tell me who should take the next task. The 3.2% figure alone doesn't give me enough to write a useful change request.