Robots.txt for your website ?

[Jack London]

Hi webmasters,

As I knew if we want Google crawl all our website then use this code in robots.txt file putted in website root directory

User-agent: *
Allow: /

but I checked some sites and they didn't put robots.txt file for their websites. Thus I want to know that it's necessary to have a robots file for my website or not ? and why ?

please share your suggestions ?

 

[Joseph_Hill]

Thanks for sharing this code, but initially I want to know why Robot.txt file should be their in our website?And why it's important in terms of SEO strategy?

 

[William]

William

Its a very good question, I have same doubt and still searching for perfect answer.

 

[ketul]

ketul

If you want to hide your sites some portion like directory and file that very secret for you, you can specify it in robots.txt, so SE crawler does not crawl and index that file and user don't see that particular page or file in search result.
Simply it tells to SE crawler that which page is allow for crawl and index and which is not.

 

[sachingoyal]

The robots.txt file is a very essential file if you wanna increase your site ranking.

Robots.txt file

When a search engine crawler comes to your website, it will look for a special root file on your website. That file is called robots.txt and it tells the search engine spider, which Web pages of your website should be indexed and which Web pages should be ignored.

Like : http://www.yourwebsite.com/robots.txt

 

[aishasony]

yes its necessary to place Robot.txt file because it get rid of all the 404 errors simply create an empty file called robots.txt and upload that to your site.

You should be able to tell if it works by examining site stats for periods before and after uploading as the number of 404 errors on the site should be dramatically reduced.

 

[aksel]

aksel

Really a good information. But I would like to know more about Robot.txt that how it actually helps in indexing.

 

[guma]

guma

Thanks But can you tell me in detail why we need robots.txt ??

 

[BloodMaster]

BloodMaster

Robots.txt was originally conceived as a way for Webmasters to prevent robots from consuming excess bandwidth, and to keep them from executing cgi scripts. However, now that the Web has gone commercial, there are many other good reasons to Disallow spiders from fetching various URLs.

 

[PankajS]

if you are confused that some website not using robot.txt then i would like to describe that robot file is only use for preventing any webpage from search engine. not to crawl webpage.

 

[JordanMcLean]

JordanMcLean

Really a good information. I was unaware of that. Can you please tell some more about robot.txt?

 

[subin]

Robots.txt is an important tool to identify crawl all websites, faster to index them on all top search engines.

 

[MarkBenson]

Mr.Jordan , you can read about robot.txt in details and for that you just have to Google it.

 

[novandak]

Hi,
I think robots.txt is most important for Search Engine.
When a search engine crawler comes to your site, it will look for a specific file on your site. That is called robots.txt. It's a simple text file.
It tells the search engine spider that which web pages should be ignored by Search Engine and which should be indexed.

 

[dagnyjbarber]

disallow term only prevent from crawling not from indexing. for prevent from indexing, you have to type noindex.

 

[kathleenortan]

Robots.txt is not always necessary, you can choose not to use them on your websites depending on the content of your website. You can use them occasionally for some websites. Some SEOs not happening practical knowledge may suggest you that it is important because they have read like this online.

 

[BloodMaster]

Through the robot.txt file Google bot/spiders/crawlers know what is in site. Crawlers get familiar with site.

 

[hoangvu]

It's necessary for you to create robots.txt when you need to protect some files or folders on your server from crawling by Google.

I found one good article on How to use Robots.txt on our webmaster sun.

Hope this helps!

 

[Deepak Kaushik]

Main purpose of uploading robots.txt file is that if you want to block any webpage, folder, file in the Google's eye then by the use of robots.txt you can do this.

 

[jasonquey]

robots.txt file in your website. The robots.txt file which presents in webmaster's will be updated automatically... No need to do any changes in webmaster's robots.txt.

 

[kathleenortan]

It is must for eCommerce business websites so as to hide their financial details.

 

[deepakrajput]

Robots.txt is the main file for Search Engine caching your site to rank them. It is set of some instructions which admin want to Allow or disallow to Search Engine.

 

[nathancs]

Robots.txt is most important for search engine spider to tell the search engine crawl the web pages.

 

[sheldonlobo]

Robots file use in many case to disallow confidential information of your website.

 

[seoindia]

There is no way that can prevent search engines or say Google to index a website and/or pages. Google will surely fetch your webpage information in any case whether you put or do not put robot.txt in your URL. Search engines will get your site information from the back links that are distributed throughout the web.

 

[websiteeonline]

Robots.txt is an important tool to identify crawl all websites, faster to index them on all top search engines.