A plain-language explanation
Robots.txt is a text file in the main website directory containing access rules for web robots to selected areas of the website.
Where it helps in practice
Robots.txt controls crawler access to paths but does not remove a URL from an index and is publicly visible.
The decision before implementation
We evaluate each rule for a named user agent and environment, avoiding accidental blocks of CSS, JS, products or the whole site.
How to verify the result
We test syntax, response code, sitemap references, bot logs and critical URLs after every release.
