Robot.txt and its usage?

hulyo

New member
Joined
Jun 15, 2017
Messages
9
Points
0
I don't understand what is the sole purpose of the robot.txt on SEO strategy, can someone explain to me?
 

SEOLosAngeles

New member
Joined
Jun 9, 2017
Messages
7
Points
0
Hi Hulyo,

Robots.txt is file that Website owners use this file to give instructions about their site to web robots & Web spider. And if you want to restrict any pages to crawl from web spider then you can use robots.txt .
 

sree31

New member
Joined
Jun 15, 2017
Messages
84
Points
0
Hi,

Robots.txt is used to tell web crawlers such as Googlebot which pages are restricted and should not be crawled. By default, google will crawl all pages on your website as long as they can be reached, that is, they are linked to by at least one other page on your website.

So you can use robots.txt to prevent crawling admin pages and other internal pages that you don't want to be indexed

Robots.txt is also useful for large websites, where you want to restrict google from crawling a sub folder or large number of pages (using simple regular expressions)
 
Last edited:

sadaykumar

New member
Joined
Jun 5, 2017
Messages
3
Points
0
Robots.txt is a file used by webmasters to tell search engine bot, which page it should crawl. Suppose you have a page in your website which you have to keep but don't want to index in search engine. You can restrict that particular Url from Robots.txt file by writing single line code: Disallow: /url of the page which you want to block
Example
Disallow: /cart
Disallow: /checkout
Disallow: /my-account

Regards
saday
 

hrishivardhan

New member
Joined
Jul 15, 2015
Messages
194
Points
0
robots.txt is like the gate of the site. it's define rules for the search spiders like which pages they can crawl and which they can not. By using this file you can create restrictions for the search engine bots and other bots so they can not crawl the pages you dont wanted to be crawled
 

jennypitula

New member
Joined
May 3, 2017
Messages
34
Points
0
Yes, I am agree with all, What is robots.txt and its use.

"I don't understand what is the sole purpose of the robot.txt"

So the reason is, to use robots.txt :

(1) Every Website have two ends: 1. Front End & 2. BackEnd

At the side of Front End we can see our website which is exist the offering/services.

But at the side of BackEnd - User can change/Edit, Modify or can delete the website code, Web Pages, Users or functionality.

All the activities having their own url some are optimized (mainly all the front end part) and some not.

When we want to promote a website at that time we don't want to show those things which is not useful from users site
Ex. abc.com/admin - This is the admin entrance url, And no one want to promote this page even don't want those types of page will appear in Search Engine (Google)

So for the above reason we block the admin part by robots.txt

(2) Some time you want to change your web page or services, at that time you have to delete those old pages or have to redirect or if you keep as it is then robots those pages so search engine will never crawl those pages.
 

Scopehosts

New member
Joined
Aug 24, 2016
Messages
59
Points
0
Web site owners use the robots.txt file to give instructions about their site to web robots. This is called The Robots Exclusion Protocol. It is a text file used to instruct the crawlers about the pages to be crawled.
 
Older threads
Replies
8
Views
2,385
Replies
5
Views
2,241
Replies
5
Views
2,342
Latest threads
Replies
2
Views
383
Replies
1
Views
366
Replies
4
Views
1,888
Replies
3
Views
794
Recommended threads
Replies
14
Views
5,655
Replies
4
Views
2,793
Replies
9
Views
10,063
Replies
13
Views
5,736

Referral contests

Referral link for :

Sponsors

Popular tags

You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.

Top