Fix the syntax of robots.txt on Vercel
Why it matters
A robots.txt that is really an html page, has rules before any User-agent line, or misspells a directive is partly or wholly ignored by crawlers. Rules you meant to apply then do nothing.
The fix
- Put the file in the public folder (public/robots.txt) so it is served as a static file. A rewrite that sends every path to index.html is the usual cause of an html answer.
- Redeploy and check https://example.com/robots.txt.
The examples use example.com. Replace it with your own domain.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Shrink the robots.txt file
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.