Allow search engines in robots.txt on WordPress
Why it matters
Googlebot and Bingbot are both disallowed from the whole site, so its pages will mostly drop out of search results, though a blocked address can still be listed without a description. That is right for a staging copy and wrong for a public site.
Staged change. Do this in stages and check each one before the next, so a mistake does not break your site or your mail.
The fix
- Confirm the block is not deliberate, then open Settings, Reading and clear Discourage search engines from indexing this site. Depending on the WordPress version this adds a noindex tag and, on older versions, a Disallow rule in robots.txt.
- Also check your SEO plugin's robots.txt editor and any physical robots.txt in the web root for a leftover Disallow line, and clear caches.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Fix the syntax of robots.txt
- Shrink the robots.txt file
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.