Seriously, you cannot edit robots.txt in Blogger hosted blogs (a.k.a blogspot). What you can do is manipulate the meta robots tag using Blogger conditional tags. One of the common problems in Blogspot is content duplication, and effective use of meta robots tag can prevent this problem.
But average blogger has almost no knowledge of blogger conditional tags. Although meta robots tag is very easy, blogger conditional tags is not simple. For example, if the following tag:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Will be added to the html source code of blogger template, all blogger posts and archives will have these meta robots tag. In the end, your blog will be completely blocked in the search engines, like an equivalent:
User-agent: *
Disallow: /
Using robotst.xt command.
Rules of preventing duplicate content in Blogger:
First rule: Of course all blogger post and the main page (front page) should be indexable, so it should have:
<META name="ROBOTS" content="INDEX,FOLLOW">
Second rule: The archived page should be placed with noindex and nofollow. Since you are not interested in search engines indexing these type of URLs since they contain just duplicates of the blogger posts.
How to implement blogger conditional tags?
Blogger conditional tags are like HTML tags with opening and close tags. For example, if you like the command to work only in the blogger front page or main page, then:
<MainPage> [HTML code here] </Mainpage>For archived page, it will be:
<ArchivePage> [HTML code here] </ArchivePage>For post URL, it will be:
<ItemPage> [ HTML code here </ItemPage>So how are you going to formulate the complete syntax that will be able to put meta robots noindex nofollow for archived, but put meta robots index follow for meta robots index follow???
The code is shown below:
<MainPage> <META name="ROBOTS" content="INDEX,FOLLOW"> </Mainpage><ArchivePage> <META name="ROBOTS" content="NOINDEX,NOFOLLOW"> </ArchivePage><ItemPage> <META name="ROBOTS" content="INDEX,FOLLOW"> </ItemPage>
How are you going to implement the above code???
Simply delete the default meta robots tag if there is and replace it with the above code. It should be placed between the
… tag. For example:
<head>
<MainPage> <META name="ROBOTS" content="INDEX,FOLLOW"> </Mainpage><ArchivePage> <META name="ROBOTS" content="NOINDEX,NOFOLLOW"> </ArchivePage><ItemPage> <META name="ROBOTS" content="INDEX,FOLLOW"> </ItemPage>
<title>Your title tag<title>
[ THE REST OF THE HEAD HMTL CODE HERE]
</head>
VERY IMPORTANT REMINDER: THIS ONLY WORKS IN BLOGGER CLASSIC TEMPLATE, NOT IN NEW XML TEMPLATE.
Very informative blog. M searching this topic since long but unable to find suitable to solve crawler errors of my two blogs newpcupdates [dot] blogspot [dot] com and sattvupdate [dot] blogspot [dot] com. I will try this trick.. Thanks for the kind info.
hey dude tell me one thing what it means User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
is this ok
It only allows Google AdSense bots to crawl the entire pages of your blog. But all bots except Google Adsense bot are not allowed to crawl search pages. Otherwise all bots are allowed to crawl non-search pages such as posts.
can the robots.txt in blogger be edited so that it also allows bots to crawl on /search ?
No. Instead use the meta robots tag noindex to control pages not to be indexed. Use blogger conditional tags to accomplish this. Good luck!
One of my blog page: http://www.healthy24.net/search/label/Health%20News is blocked. I want to open it for crawling. How to do that? Please?
And I found this Meta content:
What to change now? Please?
It cannot be indexed. Blogger intentionally blogs search pages to prevent duplicate content. You have no options but to accept that.