Allow search engines in robots.txt
Why it matters
Googlebot and Bingbot are both disallowed from the whole site, so its pages will mostly drop out of search results, though a blocked address can still be listed without a description. That is right for a staging copy and wrong for a public site.
Staged change. Do this in stages and check each one before the next, so a mistake does not break your site or your mail.
The fix
- Check that blocking search engines is not deliberate (a staging or private site, a site not yet launched). If it is deliberate, leave the file as it is and ignore this finding.
- Replace the blocking rule with an allow-all rule, or narrow it to the paths you really want to hide, and keep the Sitemap line if you have a sitemap.
- Reload https://example.com/robots.txt, then check the homepage in the URL Inspection tool of Google Search Console and request indexing.
robots.txt
User-agent: * Disallow: # Keep the next line only if the sitemap exists at this address Sitemap: https://example.com/sitemap.xml
The examples use example.com. Replace it with your own domain.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Fix the syntax of robots.txt
- Shrink the robots.txt file
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.