Difference between meta robots and robots.txt

Devin Holdcraft · Jan 15, 2017

What is the difference between meta robots and robots.txt? which should I use to block search engine spiders to access any parts on my website? are they having similar functions?

Riviera · Jan 15, 2017

I have very detail articles here for you, Sir. You can research it here:

robotstxt.org/meta.html
and
robotstxt.org/robotstxt.html

Hope that i can help.

Rob Whisonant · Jan 15, 2017

Personally I prefer to use meta tags to block a single page and robots.txt to block an entire directory.

Also note if you have links on a page that you want to send link juice but not have the page indexed, use the meta noindex tag.

Aegis software · Feb 13, 2017

If you don't want to index or follow some pages so you can use robots meta tag. But if you don't want to index a whole directory so you can use robots.txt file.

So you can restrict a whole directory or any URL format from indexing.

kinexmedia · Feb 14, 2017

I like your query . Both does the same function

If you are using Robots tags it instruct the google in different way depends on your pattern like if you write follow index it will allow spider to cache and index both . if you will write no index . follow It will cache that page but not index in google. index, no follow allow index but not cache.. index noodp - to disallow google to fetch data from directory listing

Same Robots.txt allow you to allow disallow particular page/directory/folder or entire site.

In my opinion You Should Go With Only Robots.txt file

vps9 · May 12, 2017

Robots.txt dictates site or directory wide crawl behavior, whereas meta robots can dictate indexation behavior at the individual page level.

carlbrewster · May 12, 2017

Well, Via Robot.txt file you can instruct the search engine bot how to crawl your website, you can stop the bot to crawl you private page by mentioning them in your robot.txt file, this file exists in your website root folder so, whenever bot comes to your website it will frist read instructions from robot.txt file on the other hand robot meta has to update on a particular webpage it is of 3 types

<meta name="robots" content="noindex, nofollow">
<meta name="robots" content="nofollow, Index">
<meta name="robots" content="index, nofollow">

nickchernets · May 19, 2017

There are slightly different uses for each.

personally, I try and use the meta robots where possible, search engines seem to follow these directives more strongly than the robots.txt, you can also choose options such as noindex,follow which is really helpful for certain areas.

The problem I've got with the robots.txt is that it's too easy to make a mistake and harm a lot of your site, and you're completely blocking off areas of the site, so if there's links in there that you want to get indexed but forget about the robots.txt you'll have a problem, whereas if you use noindex,follow that's not an issue.

Difference between meta robots and robots.txt

Devin Holdcraft

New member

Riviera

New member

Rob Whisonant

Moderator

KarenByerly

Member

Rob Whisonant

Moderator

yunarel

New member

Rob Whisonant

Moderator

Aegis software

New member

kinexmedia

New member

vps9

New member

carlbrewster

New member

nickchernets

New member

Latest posts New threads

Latest posts

New threads

Referral contests

Refer Your Friends to WebmasterSun to Win Cash and Prizes!

Latest Offers New Reviews

Latest affiliate promotion

Popular Contributors - Past 30 Days

Sponsors

Popular tags