Freqo Compass · Guide
Sitemap and robots.txt
These two small files tell search engines what to read on your site. A mistake in either one can hide whole sections of your site from search results.
Why it matters
- robots.txt tells robots what they may read. One wrong line, such as “Disallow: /”, can block your whole site.
- The sitemap lists the pages you want to appear in search results. Outdated addresses (redirects, errors, pages marked noindex) send mixed signals.
How to check them
Open yoursite.com/robots.txt and yoursite.com/sitemap.xml in your browser, and check that the sitemap is declared in robots.txt and in Google Search Console. Freqo Compass reads both files each week and compares them with the pages it finds on your site.
How to fix them
- Make sure robots.txt does not block pages you want in search results, and declare your sitemap in it (Sitemap: https://yoursite.com/sitemap.xml).
- List only final, indexable addresses in the sitemap: no redirects, no errors, no noindex pages.
- Keep the sitemap up to date. Most site builders and CMSs can generate it for you.
- For similar pages, a canonical tag points to the main version. For multilingual sites, hreflang tags must link every language version to the others, both ways.
What Freqo Compass checks
- A robots.txt that blocks the whole site
- A sitemap that is missing, unreadable or lists wrong addresses
- Pages that are missing from the sitemap
- Canonical tags that are broken or point to another site
- hreflang tags without a return link
- Pages marked noindex
This is one of the 39 checks Freqo Compass runs on your site every week, each with an explanation of how to fix it. Discover Freqo Compass
Other guides
- Broken links and 404 errors
- Page titles and meta descriptions
- Real page speed (Core Web Vitals)
- Images without alternative text
- Canonical tags
- HTTPS, www and redirects
- hreflang and page language
No tool can guarantee a position in search results, and neither do we.