0
Remove: Disallow: /search

From where can I remove this line? I am using Blogger and in webmaster it shows many of my pages are blocked. From the following Q&A, I know that I need to remove this text but don't know from where.

Blocked urls by Google

Improve this question
2

From https://forums.digitalpoint.com/threads/how-to-edit-robots-txt-in-blogspot-blog.422059/

You cant change the robots.txt file in Blogger unless you host your own blog in your server.

As a general rule, "search" pages should be disallowed in robots.txt. Google does not want to include pages of search results in the index. If they did so, users would be clicking from one set of search results to another. Google does not consider that good user experience. In fact, Google has been known to de-index (penalize) entire sites that allow their site search results to be indexed.

Improve this answer
  • thanks for the reply....but then how can i remove the blocked url's which are being shown as blocked... – utsav Mar 16 '13 at 17:42
  • Search pages should be blocked. You can't remove them as long as you are using blogger. – Stephen Ostermiller♦ Mar 17 '13 at 0:16
1

Your can change/add custom robots.txt on Blogger. Go to Dashboard > Settings › Search preferences > Custom robots.txt >, click edit to enable and edit.

To remove Disallow: /search only. Fill robots.txt format below to text-area (change http://yourdomain.com to your homepage URL):

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Allow: /

Sitemap: http://yourdomain.com/feeds/posts/default?orderby=UPDATED

INFO: if you remove Disallow: /search from your robots.txt your website will experience possible problems with duplicate meta titles and descriptions which can be bad for SEO.

Improve this answer
0

Blogger now provides the ability to edit the robots.txt file - although it doesn't feel like you are editing a file.

Log into the Blogger dashboard, and go to Options > Search. There is an edit link beside robots.txt in that screen.

Improve this answer
  • can u plz tell me what to write in that robots file, so that i can reduce the number of blocked urls. my site is housefullhub.blogspot.in – utsav Mar 16 '13 at 17:44
  • 1
    You seem to be missing the point that you should not remove the Disallow: /search rule. Search URLs should be blocked from indexing so that only real content pages are indexed. – Andrew Lott Nov 1 '17 at 23:14
-1

The robots.txt file is always found in the root folder of your site.

You can usually edit it using a file-manager which cPanel nearly always has. If your site does not have cPanel, there is probably something similar but I cannot guess without knowing more.

The nearly universal way is via FTP. You connect to your server, download the file. Edit it and then upload it back. If you use an IDE such as DreamWeaver for your web-development that is what it usually does. You can use any FTP package though, including command-line ones bundled with most operating systems.

There are two things to note:

  1. When using someone else's server, they can block or override access to pretty much anything. If for some reason, you cannot do the change yourself you will have to file a ticket request with your hosting provider.
  2. Google does not want to index anybody else's search results. For this reason, you will often find search pages blocked in the robots.txt. This may be why its there in the first place. If it just happens to be called search because you have pages about search rather than generated by a search feature, then you should remove it.
Improve this answer
  • 1
    While the the robots.txt file is always found in the root folder of your site, not all CMS's allow direct access to this file. In particular Blogger does not allow this. – MaryC.fromNZ Mar 15 '13 at 15:24
  • @MaryC.fromNZ - That is what is says in point 1 under things to note! – Itai Mar 15 '13 at 15:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.