Robots.txt Rules
& Crawl Validator

Test disallow directories and crawl rules against active bots. Verify if search indexers or AI data-scrapers can crawl specific URL paths.

1

Fetch Robots.txt

2

Test Scraper Directives

3

Rule Evaluation Results

No check performed yet.

Click 'Test Robots Directives' to run validation checks.

4

Robots.txt Editor & Rules Source

Editable

You can customize these rules manually to test hypothetical block configurations instantly.