Robots.txt Generator

Search Engine Optimization

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

The file has instructions regarding how to crawl your website. Websites use this robot exclusion protocol to instruct the bots that are to be indexed on the website. You can limit these areas in the website that can be processed or not by these crawlers. Without this Robot’s txt file, the search engine crawlers do not index the pages on the website. It may be a small file but help you to rank your page. It contains “Allow”, “DisAllow” below the user-agent.

Why is it Needed?

When you try to create this file it takes time and you are in confusion to add the Allow and DisAllow on the site. This tool helps you to generate the file automatically by feeding the right information about your website. Otherwise, you should be careful to know the guidelines to create Robots txt files like Crawl-delay, Allow, and DisAllow.