Google’s Gary Illyes shares an unconventional but valid method for centralizing robots.txt rules on CDNs.
- Robots.txt files can be centralized on CDNs, not just root domains.
- Websites can redirect robots.txt from main domain to CDN.
- This unorthodox approach complies with updated standards.