How to Audit Whether Your Site Is in Common Crawl (5-Minute LLM...

Yes—you can check in about five minutes whether your domain appears in Common Crawl’s public index, the upstream web archive that feeds many LLM pre-training corpora. Query the CDX Index Server with your domain pattern. JSON lines with url, timestamp, status, and filename mean CCBot captured pages in that monthly crawl. An empty response means your site was not in that snapshot—an upstream visibility problem that on-page SEO alone cannot fix.

90-second walkthrough: curl the latest CDX index, read JSON captures, and interpret a negative result. Full transcript is in the captions file.

Key Takeaways

Common Crawl is not Google Search, and it is not proof your pages trained GPT-4 or Claude. It is the first public gate you can audit without vendor access.

Common Crawl is the largest open web corpus. Mozilla’s Roots analysis found that a large share of major LLMs draw from filtered Common Crawl derivatives. Derivatex’s eligibility guide cites Mozilla data showing GPT-3 used more than 80% filtered CC tokens.
The 5-minute audit uses the CDX API. Pick the latest crawl from collinfo.json, then run: curl "https://index.commoncrawl.org/CC-MAIN-2026-25-index?url=YOURDOMAIN.com/*&output=json" | head. We verified docusign.com returns 200-status HTML captures in July 2026; convertos.ai returned no captures in that crawl—a real troubleshooting example.
In Common Crawl ≠ in the model. Raw CC snapshots pass through filters (C4, FineWeb, Dolma, WanJuan-CC). Podrez’s audit write-up notes FineWeb-Edu rejected about 92% of CC data. Presence is eligibility, not inclusion.
CCBot robots.txt is the upstream switch. Blocking CCBot with Disallow: / removes you from future CC snapshots by design. Some publishers do this on purpose; growth and GEO teams should treat it as a conscious tradeoff.
Next layer after CC: citation visibility in live AI answers. Run a free AI visibility snapshot once crawl eligibility looks healthy, then track which URLs get cited.

Common Crawl to LLM training data pipeline infographic

Figure: From CCBot crawl to filtered training corpora. “In CC” is step 2—not the finish line.

What Is a Common Crawl LLM Training Data Audit?

A Common Crawl audit checks whether your site’s pages appear in Common Crawl’s monthly WARC snapshots—the open archive many LLM datasets start from. You are not asking “did OpenAI train on my homepage?” (no vendor publishes that). You are asking “was my site reachable by CCBot and stored in a public crawl?” That is the baseline condition for CC-derived training data. Promptly Found’s June 2026 field-guide summary frames this as upstream GEO: before RAG retrieval, before schema work, before link building, the crawler has to reach the page. If the page never enters a CC snapshot, it cannot flow into downstream filtered corpora that models actually train on.

Term	Plain definition	What the audit proves
Common Crawl	Non-profit monthly web crawl published on AWS	Your pages were archived in a given crawl ID
CCBot	Common Crawl’s crawler user-agent	robots.txt and server access allowed the bot
CDX index	URL-level index to WARC captures	You can query captures without downloading petabytes
WARC file	Web archive container for HTTP responses	A `filename` field in JSON points to the stored page
Filtered corpus	Cleaned subset (FineWeb, C4, etc.)	Not verified by the CDX query alone
LLM training	Model pre-training on text tokens	Not directly observable for closed models

This audit fits B2B SaaS, publishers, and ecommerce teams who want a measurable “AI data footprint” check without guessing. It pairs naturally with technical SEO: if Googlebot can crawl but CCBot cannot, you may rank in Google while remaining absent from CC-based training pipelines. For Convertos readers, think of it as layer zero in a GEO hub stack: crawl eligibility → training-corpus eligibility (inferred) → live citation visibility → conversion from AI referrals.

How the Common Crawl → LLM Pipeline Works

Common Crawl publishes a new crawl roughly every month (IDs like CC-MAIN-2026-25). CCBot fetches a sample of the web—not every URL on every site—writes responses into WARC files, and exposes a CDX index so anyone can look up captures by URL pattern. Research labs and vendors download these archives or buy pre-filtered derivatives to build training sets. The official CDXJ documentation describes three practical access paths: the Index Server UI, curl against the CDX API, and libraries like cdx_toolkit for Python workflows.

Figure: index.commoncrawl.org—select a crawl archive, enter yourdomain.com/*, click Search.

End-to-end flow

Crawl — CCBot requests URLs allowed by robots.txt and server rules.
Archive — Successful responses land in WARC segments on AWS Open Data.
Index — CDX rows map url → timestamp, status, mime, filename.
Filter — Curators apply language, quality, dedup, and safety filters (see Hugging Face FineWeb pipeline docs).
Train — Model vendors mix CC-derived text with books, code, and proprietary data.

Your audit sits at steps 1–3. Steps 4–5 are opaque for closed models, which is why honest articles say “eligible” rather than “trained.”

What JSON fields mean

When the API returns a line like this (from our July 2026 docusign.com test):

{"url": "https://www.docusign.com/", "timestamp": "20260607080852", "status": "200", "mime": "text/html", "filename": "crawl-data/CC-MAIN-2026-25/segments/.../warc/....warc.gz"}

Field	How to read it
`timestamp`	Capture time in UTC (`YYYYMMDDHHMMSS`)
`status`	HTTP status at crawl time (`200` = page fetched)
`mime`	Detected content type
`filename`	Path to the WARC record (proof of archive)
`languages`	Language guess when available

Check three recent crawls, not one. A site can miss April yet appear in June if CCBot finally reached it—or if you fixed a robots block.

5-Minute Common Crawl Audit Checklist

Run this checklist before you spend budget on content rewrites for “AI training visibility.” Total time: about five minutes if you have terminal access and your domain’s robots.txt handy.

Step	Action	Pass signal	Fail signal
1	Open `https://yoursite.com/robots.txt`	`CCBot` not blocked (or intentional block documented)	`User-agent: CCBot` + `Disallow: /`
2	List latest crawl ID	`curl -s https://index.commoncrawl.org/collinfo.json \	head`	Cannot reach index server
3	Query domain in newest crawl	JSON lines with `status: 200` for key URLs	`"No Captures found"` or empty body
4	Repeat for 2 older crawls	At least one crawl shows homepage or `/blog/`	Zero captures across 3 months
5	Spot-check WAF/CDN	CCBot not challenged separately from Googlebot	Bot fight returns 403 only for CCBot
6	Log results	Spreadsheet row: crawl ID, URL count, sample URLs	No baseline for retest

Copy-paste commands

Replace YOURDOMAIN.com and the crawl ID with values from step 2.

# Latest crawl collections
curl -s "https://index.commoncrawl.org/collinfo.json" | python3 -m json.tool | head -30

# Domain audit (JSON)
curl -s "https://index.commoncrawl.org/CC-MAIN-2026-25-index?url=YOURDOMAIN.com/*&output=json" | head -20

# Homepage only
curl -s "https://index.commoncrawl.org/CC-MAIN-2026-25-index?url=YOURDOMAIN.com/&output=json&limit=5"

Source tip: Chris Nectiv Common Crawl curl demo

Figure: Social tip that matches the official CDX workflow—query domain.com/*, read JSON lines.

If you get zero captures

Work through this order:

Robots — Remove accidental CCBot block; keep block only if legal/compliance requires it.
Server logs — Search for CCBot hits and 403/503 patterns.
CDN bot rules — Cloudflare/Akamai “verified bot” lists sometimes omit CCBot.
Site age — Brand-new domains may simply wait for the next monthly crawl.
Retest in 30–45 days — CC does not crawl your site daily.

Use Convertos robots validation in the toolkit alongside manual checks when multiple AI bots (GPTBot, ClaudeBot, PerplexityBot) matter for your policy.

How to Measure and Retest Common Crawl Visibility

Treat Common Crawl like an infrastructure monitor, not a one-time vanity check. You want a small dashboard that answers: “Are we still crawlable upstream?” and “Is coverage improving for money pages?”

Metric	Definition	Cadence	Tool
CC capture rate	% of priority URLs with ≥1 `200` capture in last 3 crawls	Monthly	CDX API script
Crawl freshness	Newest `timestamp` for homepage	Monthly	CDX query
robots drift	Unexpected `CCBot` rule changes	Weekly	Diff on robots.txt
Page depth in CC	Blog/docs URLs indexed vs sitemap priority	Quarterly	CDX + sitemap list
Negative crawl rate	`403`/`5xx` rows in CDX	Monthly	CDX status field
Downstream proxy	Brand prompts in ChatGPT/Perplexity	Bi-weekly	AI visibility checker

Scoring rubric (0–100)

Score	Criteria
80–100	Homepage + core product/docs URLs in 3/3 recent crawls; CCBot allowed; no WAF blocks
50–79	Homepage in ≥2 crawls; some section gaps (e.g., `/pricing/` missing)
20–49	Only scattered URLs or redirects; likely partial crawl sample
0–19	Zero captures across 3 crawls or deliberate CCBot block

Sample tracking row

Month	Crawl ID	Homepage	`/pricing/`	`/blog/*` count	Notes
2026-07	CC-MAIN-2026-25	200	200	42 URLs	Baseline
2026-06	CC-MAIN-2026-21	200	—	38 URLs	Pricing missed one crawl
2026-05	CC-MAIN-2026-17	301→200	200	35 URLs	HTTP redirect captured

Automate the CDX pull with a cron job or CI step—do not hammer the public index server with full-site wildcard bulk jobs. The Index Server help text asks users to use Columnar Index downloads for TLD-scale work. After CC scores stabilize upward, shift measurement to citation and referral signals in your GEO workflow. Training-data eligibility and live answer visibility are related but not identical problems.

Common Mistakes (and How to Fix Them)

Mistake	Why it misleads	Fix
“No JSON = my SEO is broken”	CC samples the web; absence in one crawl ≠ Google deindex	Check 3 crawls + Google Search Console
“JSON = ChatGPT knows my brand”	Training filters drop most CC pages	Add prompt tests for live answers
Testing only the homepage	CC may capture `/blog/` but not `/`	Query `domain.com/*` and top 20 URLs
Using an expired crawl ID	Old indexes rotate off the server	Read `collinfo.json` first
Ignoring `301`/`404` rows	CDX stores non-200 captures too	Filter `status:200` in your script
Blocking CCBot “for security”	Removes you from CC by design	Separate security policy from GEO policy
Bulk-querying `*.com` from the API	Violates index server fair use	Use Columnar Index downloads
One-language sites skipping `languages` field	Helps confirm correct locale capture	Log `languages` from JSON
Expecting daily recrawls	CC is monthly-ish	Retest on crawl release notes

Real negative example: convertos.ai in CC-MAIN-2026-25

Our live July 2026 query returned "No Captures found for: convertos.ai/" in the latest crawl while docusign.com returned many captures. That does not automatically mean Convertos has bad SEO—it means we were not in that CC snapshot yet. The honest response is: monitor robots, wait for subsequent crawls, and prioritize pages you need in public corpora. This is the same nuance Ewelina Podrez stresses: missing one URL in one crawl does not prove “AI invisibility,” but missing all priority URLs across multiple crawls is a red flag worth fixing.

FAQ

These questions match People Also Ask, related questions, and community source signals from our SERP research (including the Chris Nectiv/X tip).

How do I check if my website is in Common Crawl?

Query the CDX index with yourdomain.com/*; rows with status: 200 were archived in that crawl.

Does being in Common Crawl mean my site trained an LLM?

No. Vendors do not publish per-domain training lists. CC presence only shows the public crawl gate passed; filtered datasets may still drop your pages (Derivatex).

What is CCBot in robots.txt?

CCBot is Common Crawl’s crawler. Blocking it with Disallow: / opts out of future CC snapshots. That is valid for publishers who do not want open-archive inclusion.

Is Common Crawl the same as Google indexing?

No. Google uses its own crawlers and index for Search and AI Overviews. You can rank in Google and still be absent from a given CC crawl, and vice versa.

How often does Common Crawl crawl my site?

Roughly monthly releases, but each pass captures a subset of URLs—not a full site mirror. See the CDX collection list for dates.

Can I remove my site from Common Crawl after it was crawled?

Contact and policy options are described on commoncrawl.org. Prevention via robots is easier than retroactive removal.

What should I do after a successful CC audit?

Run live AI visibility tests on your priority prompts, fix citation gaps on key URLs, and schedule a technical page audit for structured data and fetch quality.

How to Audit Whether Your Site Is in Common Crawl (5-Minute LLM Training Data Check)