How to Add Custom Robots.txt file in Blogger
In our previous, we had discussed about How to add Custom Robots Header Tags for blogger and here were back with another great and most important Post for all of you. Today i am going to enable a custom Robots.txt file in Blogger. which can really help you to index your blog in Google faster and it will also make your blog more search engine friendly (SEO). Robots.txt file is used in every website but in the past with old Blogger interface we're unable to add that file in our blogs but now in the new interface which is really amazing with a lot of great features we can enable that file in Blogger. Those who don't know about Robots.txt file can read the below passage to understand that file.
-----------------------------------------------------------------------------------------------------------------------------
What is Robots.txt?
-----------------------------------------------------------------------------------------------------------------------------
Robots.txt is a text file which contains few lines of simple code. It is saved on the website or blog’s server which instruct the web crawlers to how to index and crawl your blog in the search results. That means you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog labels page, your demo page or any other pages that are not as important to get indexed. Always remember that search crawlers scan the robots.txt file before crawling any web page.
-----------------------------------------------------------------------------------------------------------------------------
How to Add Custom Robots.txt file in Blogger
-----------------------------------------------------------------------------------------------------------------------------
1. Sign in to your blogger Account. (Click Here)
2. Now Navigate the Settings >> Search Preferences
3. Now Click on Edit Link Under the Custom " Robots.txt " section as shown Below screenshot.
4. Now a check box will appear tick mark "Yes" and a box will appear where you have to write the robots.txt file. So if you want to use our file then copy the code below and paste it in that box.
User-agent: Mediapartners-Google
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /
Sitemap: http://www.yourdmomain.com/feeds/posts/default?orderby=updated
5. Note : The first line "User-agent: Mediapartners-Google" is for Google AdSense. So if you are using Google AdSense in your blog then remain it same otherwise remove it.
6. Now Click "Save Changes" Button.
7. Custom Robots.txt file in Blogger is Done !.
-----------------------------------------------------------------------------------------------------------------------------Explanation
----------------------------------------------------------------------------------------------------------
- User-agent: Mediapartners-Google : So, This is a first command which is for those blogs which are using Google AdSense if you are not using Google AdSense then remove it. In this command, we're telling the AdSense's separate robot that crawl all pages which are having AdSense Ads.
- User-agent: * : Here the User-agent is calling the robot and * is for all the search engine's robots like Google, Yahoo etc.
- Disallow: /search?q=* : This line tells the search engine's crawler not to crawl the search pages.
- Disallow: /*?updated-max=* : This one disallows the search engine's crawler to do not index or crawl label or navigation pages.
- Allow: / : This one allows to index the homepage or your blog.
- Sitemap : So this last command tells the search engine's crawler to index the every new or updated post.