Lawscan.org Scan free

robots.txt configured

Accessibility & Usability · Check 48/55 · −1 point · All websites

What it means

A /robots.txt file tells crawlers which areas to avoid and where the sitemap is.

Why it matters

It keeps private areas out of search results and avoids wasted crawling.

Legal and technical reference: Robots Exclusion Protocol (RFC 9309)

How to pass this check

Quick fix: Publish /robots.txt with your crawl rules and your sitemap URL.

  1. Publish /robots.txt at the root of the domain.
  2. Disallow admin, cart and search pages as appropriate.
  3. Add the Sitemap: line.

How Lawscan checks it

The scanner downloads /robots.txt and checks that it contains valid rules (a soft 404 page does not count).

Does your website pass this check?

Scan it now: you get the result of all 55 checks in a few seconds.

Scan my website free

Other checks in this category

← Back to the full Compliance Index

Automated, indicative information. It is not legal advice.

Scan your website free →