robots.txt configured
What it means
A /robots.txt file tells crawlers which areas to avoid and where the sitemap is.
Why it matters
It keeps private areas out of search results and avoids wasted crawling.
Legal and technical reference: Robots Exclusion Protocol (RFC 9309)
How to pass this check
Quick fix: Publish /robots.txt with your crawl rules and your sitemap URL.
- Publish /robots.txt at the root of the domain.
- Disallow admin, cart and search pages as appropriate.
- Add the Sitemap: line.
How Lawscan checks it
The scanner downloads /robots.txt and checks that it contains valid rules (a soft 404 page does not count).
Does your website pass this check?
Scan it now: you get the result of all 55 checks in a few seconds.
Other checks in this category
- Color contrast test −1
- Font readability test −1
- Mobile-friendly navigation −2
- Page load <3 sec −2
- XML sitemap present −1
- Image ALT attributes properly filled −1
- Keyboard navigation functional −1
← Back to the full Compliance Index
Automated, indicative information. It is not legal advice.