Robot.txt and its usage?

[hulyo]

I don't understand what is the sole purpose of the robot.txt on SEO strategy, can someone explain to me?

 

[SEOLosAngeles]

Hi Hulyo,

Robots.txt is file that Website owners use this file to give instructions about their site to web robots & Web spider. And if you want to restrict any pages to crawl from web spider then you can use robots.txt .

 

[sree31]

Hi,

Robots.txt is used to tell web crawlers such as Googlebot which pages are restricted and should not be crawled. By default, google will crawl all pages on your website as long as they can be reached, that is, they are linked to by at least one other page on your website.

So you can use robots.txt to prevent crawling admin pages and other internal pages that you don't want to be indexed

Robots.txt is also useful for large websites, where you want to restrict google from crawling a sub folder or large number of pages (using simple regular expressions)

 

[sadaykumar]

Robots.txt is a file used by webmasters to tell search engine bot, which page it should crawl. Suppose you have a page in your website which you have to keep but don't want to index in search engine. You can restrict that particular Url from Robots.txt file by writing single line code: Disallow: /url of the page which you want to block
Example
Disallow: /cart
Disallow: /checkout
Disallow: /my-account

Regards
saday

 

[hrishivardhan]

robots.txt is like the gate of the site. it's define rules for the search spiders like which pages they can crawl and which they can not. By using this file you can create restrictions for the search engine bots and other bots so they can not crawl the pages you dont wanted to be crawled

 

[jennypitula]

Yes, I am agree with all, What is robots.txt and its use.

"I don't understand what is the sole purpose of the robot.txt"

So the reason is, to use robots.txt :

(1) Every Website have two ends: 1. Front End & 2. BackEnd

At the side of Front End we can see our website which is exist the offering/services.

But at the side of BackEnd - User can change/Edit, Modify or can delete the website code, Web Pages, Users or functionality.

All the activities having their own url some are optimized (mainly all the front end part) and some not.

When we want to promote a website at that time we don't want to show those things which is not useful from users site
Ex. abc.com/admin - This is the admin entrance url, And no one want to promote this page even don't want those types of page will appear in Search Engine (Google)

So for the above reason we block the admin part by robots.txt

(2) Some time you want to change your web page or services, at that time you have to delete those old pages or have to redirect or if you keep as it is then robots those pages so search engine will never crawl those pages.

 

[Scopehosts]

Web site owners use the robots.txt file to give instructions about their site to web robots. This is called The Robots Exclusion Protocol. It is a text file used to instruct the crawlers about the pages to be crawled.