Robots.txt Generator

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

The file has instructions regarding how to crawl your website. Websites use this robot exclusion protocol to instruct the bots that are to be indexed on the website. You can limit these areas in the website that can be processed or not by these crawlers. Without this Robot’s txt file, the search engine crawlers do not index the pages on the website. It may be a small file but help you to rank your page. It contains “Allow”, “DisAllow” below the user-agent.

Why is it Needed?

When you try to create this file it takes time and you are in confusion to add the Allow and DisAllow on the site. This tool helps you to generate the file automatically by feeding the right information about your website. Otherwise, you should be careful to know the guidelines to create Robots txt files like Crawl-delay, Allow, and DisAllow.