Shrink the robots.txt file on WordPress
Why it matters
Google reads only the first 500 KiB of robots.txt and ignores the rest, so rules at the end may never apply. Very long files are also hard to maintain.
The fix
- Open the robots.txt editor of your SEO plugin, or the physical robots.txt in the web root, and remove stale rules left by old plugins.
- Prefer noindex settings in the SEO plugin for individual pages over long Disallow lists.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Fix the syntax of robots.txt
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.