Crawl and Index Checks: From Status Codes to Sitemaps
Why this matters
Two parts can look identical in a catalogue yet behave differently on site. Organising the facts in a checkable order reduces rework and repeated questions.
01 / Check crawlability first
robots.txt, server responses, redirect chains and rendering decide whether a page can be collected at all. Verify with a crawler before anything else.
02 / Then check indexation
Crawled is not indexed. Review canonical tags, noindex, duplicate pages and parameter URLs against the real page inventory.
03 / Finally audit the sitemap and internal links
List only indexable canonical URLs, and make sure key pages have clickable internal entry points instead of living only in XML.
Content here supports lookup and reading. Always confirm against the latest supplier datasheet and official publications.
Put this on your checklist
- Export crawl errors with status codes
- Compare indexed pages with the page inventory
- Remove non-canonical URLs from the sitemap
Every item can be checked on its own without extra tooling. Record the result in one table with reviewer, date and source document for later review.