Purple FarmApps and tools

Robots.txt Generator

Configure and generate robots.txt for your site

Use * for all crawlers

Generated robots.txt

# Click "Generate robots.txt" to see output

About robots.txt

  • β€’ Place robots.txt in your website root directory
  • β€’ Use Allow: / to allow all paths
  • β€’ Use Disallow: /path/ to block specific paths
  • β€’ Crawl-delay sets wait time between requests
  • β€’ Include sitemap URL for better indexing