Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
Remove: Disallow: /search 

From where can I remove this line? I am using Blogger and in webmaster it shows many of my pages are blocked. From the following Q&A, I know that I need to remove this text but don't know from where.

Blocked urls by Google

share|improve this question
add comment

4 Answers

From https://forums.digitalpoint.com/threads/how-to-edit-robots-txt-in-blogspot-blog.422059/

You cant change the robots.txt file in Blogger unless you host your own blog in your server.

As a general rule, "search" pages should be disallowed in robots.txt. Google does not want to include pages of search results in the index. If they did so, users would be clicking from one set of search results to another. Google does not consider that good user experience. In fact, Google has been known to de-index (penalize) entire sites that allow their site search results to be indexed.

share|improve this answer
    
thanks for the reply....but then how can i remove the blocked url's which are being shown as blocked... –  utsav Mar 16 '13 at 17:42
    
Search pages should be blocked. You can't remove them as long as you are using blogger. –  Stephen Ostermiller Mar 17 '13 at 0:16
add comment

Blogger now provides the ability to edit the robots.txt file - although it doesn't feel like you are editing a file.

Log into the Blogger dashboard, and go to Options > Search. There is an edit link beside robots.txt in that screen.

share|improve this answer
    
can u plz tell me what to write in that robots file, so that i can reduce the number of blocked urls. my site is housefullhub.blogspot.in –  utsav Mar 16 '13 at 17:44
add comment

Your can change/add custom robots.txt on Blogger. Go to Dashboard > Settings › Search preferences > Custom robots.txt >, click edit to enable and edit.

To remove Disallow: /search only. Fill robots.txt format below to text-area (change http://yourdomain.com to your homepage URL):

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Allow: /

Sitemap: http://yourdomain.com/feeds/posts/default?orderby=UPDATED

INFO: if you remove Disallow: /search from your robots.txt your website will experience possible problems with duplicate meta titles and descriptions which can be bad for SEO.

share|improve this answer
add comment

The robots.txt file is always found in the root folder of your site.

You can usually edit it using a file-manager which cPanel nearly always has. If your site does not have cPanel, there is probably something similar but I cannot guess without knowing more.

The nearly universal way is via FTP. You connect to your server, download the file. Edit it and then upload it back. If you use an IDE such as DreamWeaver for your web-development that is what it usually does. You can use any FTP package though, including command-line ones bundled with most operating systems.

There are two things to note:

  1. When using someone else's server, they can block or override access to pretty much anything. If for some reason, you cannot do the change yourself you will have to file a ticket request with your hosting provider.
  2. Google does not want to index anybody else's search results. For this reason, you will often find search pages blocked in the robots.txt. This may be why its there in the first place. If it just happens to be called search because you have pages about search rather than generated by a search feature, then you should remove it.
share|improve this answer
1  
While the the robots.txt file is always found in the root folder of your site, not all CMS's allow direct access to this file. In particular Blogger does not allow this. –  MaryC.fromNZ Mar 15 '13 at 15:24
    
@MaryC.fromNZ - That is what is says in point 1 under things to note! –  Itai Mar 15 '13 at 15:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.