1

Here is content of my robots.txt file:

User-agent: *
Disallow: /images/
Disallow: /upload/
Disallow: /admin/

As you can see, I explicitly disallowed all robots to index the folders images, upload and admin. The problem is that one of my clients sent request for removing the content from the images folder because .pdf document from the images folder appeared in the google search results. Can anyone explain me what I'm doing wrong here, and why google indexed my folders?

| improve this question | |
  • In addition, I would recommend you to use the "Remove URLs" feature on Google Webmaster Tools and submit a removal request of your /images/ folder from Google's search results. – user43821 Sep 11 '14 at 6:16
  • Same question was asked on SO previously. – unor Sep 13 '14 at 12:44
5

Google isn't ignoring your robots.txt as robots.txt does not tell Google not to list that content in its search results. It tells Google not to crawl content. So Googlebot cannot find this content on its own. But if it does find this content it will still list it in the search results.

To keep content out of the search results you need to use the x-robots-tag HTTP header. It tells search engines not to index that content. If you're using Apache you can place a file called .htaccess in each of the mentioned directories with the following line in it:

X-Robots-Tag: noindex

That will prevent the content of those directories to not appear in the search results.

| improve this answer | |
  • With all due respect of course. You may need to re-read the page you linked to. So, if a URL is disallowed in your robots.txt, that URL and its contents won't appear in Google Search results. This was and remains the standard. I know that Google has been creating links in their SERPs for resources that are protected by robots.txt lately. Google likes to re-write the rules. However, they do disappear again as far as I know. I like the .htaccess solution you provided. Again- with respect. – closetnoc Sep 10 '14 at 17:02
  • 2
    @closetnoc While Google won't crawl or index the content blocked by robots.txt, we might still find and index information about disallowed URLs from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the site can still appear in Google search results. That's from the page I linked to. There's another place (on another page) where Google mentions they do this because sites like the CA DMV blocks their sites using robots.txt but Google wants to show their results as they are commonly searched for. – John Conde♦ Sep 10 '14 at 17:12
  • I'll see if I can find that link after lunch for reference. – John Conde♦ Sep 10 '14 at 17:13
  • Found it. – John Conde♦ Sep 10 '14 at 17:16
  • Okay. Cool. That is what I meant by I know that Google has been creating links in their SERPs for resources that are protected by robots.txt lately. It has been my experience that these disappear, however, if a long standing robots.txt rule exists, Google should know better even though it does not appear that they do. This seems to be a new(ish) behavior (within the past couple of years). Thanks! – closetnoc Sep 10 '14 at 17:22
-1

While Google won’t crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site can appear in Google search results.

User-agent: *
Disallow:/images/

According to the above rule, all robots are denied access to the /images/ folder of a site. Additionally, a specific rule can be set to explicitly disallow access to all files within a folder:

Disallow:/images/*

| improve this answer | |
  • 1
    /images/ and /images/* are the same thing. The path specified in a Disallow rule is simply a prefix. (Therefore a "wildcard" * is never required at the end of the path.) – MrWhite Sep 10 '14 at 13:07
  • This is incorrect. Google will still list content blocked by robots.txt. – John Conde♦ Sep 10 '14 at 13:07
  • To remove all the images on your site from our index, place the following robots.txt file in your server root: User-agent: Googlebot-Image Disallow: / – Ipsita Sep 10 '14 at 13:19
  • Google has introduced increased flexibility to the robots.txt file standard through the use asterisks. Disallow patterns may include "*" to match any sequence of characters, and patterns may end in "$" to indicate the end of a name. – Ipsita Sep 10 '14 at 13:25
  • To remove all files of a specific file type (for example, to include .jpg but not .gif images), you'd use the following robots.txt entry: User-agent: Googlebot-Image Disallow: /*.gif$ By specifying Googlebot-Image as the User-agent, the images will be excluded from Google Image Search. Refer: support.google.com/webmasters/answer/35308?hl=en – Ipsita Sep 10 '14 at 13:26
-3

No Allow code? Try add 

User-agent: *
Allow: /
Disallow: /images/
Disallow: /upload/
Disallow: /admin/
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.