Fix the syntax of robots.txt on WordPress
Why it matters
A robots.txt that is really an html page, has rules before any User-agent line, or misspells a directive is partly or wholly ignored by crawlers. Rules you meant to apply then do nothing.
The fix
- If a plugin edits robots.txt, open its editor (usually under the SEO plugin's tools) and fix the lines there. A physical robots.txt in the web root overrides the virtual one.
- Reload https://example.com/robots.txt to check the result.
The examples use example.com. Replace it with your own domain.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Shrink the robots.txt file
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.