Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
Here is content of my robots.txt file:
User-agent: *
Disallow: /images/
Disallow: /upload/
Disallow: /admin/
As you can see, I explicitly disallowed all robots to index the folders images, upload and admin. The problem is that one of my clients sent request for removing the content from the images folder because .pdf document from the images folder appeared in the google search results. Can anyone explain me what I'm doing wrong here, and why google indexed my folders?
Google isn't ignoring your robots.txt as robots.txt does not tell Google not to list that content in its search results. It tells Google not to crawl content. So Googlebot cannot find this content on its own. But if it does find this content it will still list it in the search results.
To keep content out of the search results you need to use the x-robots-tag HTTP header. It tells search engines not to index that content. If you're using Apache you can place a file called .htaccess in each of the mentioned directories with the following line in it:
X-Robots-Tag: noindex
That will prevent the content of those directories to not appear in the search results.
While Google won't crawl or index the content blocked by robots.txt, we might still find and index information about disallowed URLs from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the site can still appear in Google search results. That's from the page I linked to. There's another place (on another page) where Google mentions they do this because sites like the CA DMV blocks their sites using robots.txt but Google wants to show their results as they are commonly searched for.
– John Conde♦
Sep 10 '14 at 17:12
While Google won’t crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site can appear in Google search results.
User-agent: *
Disallow:/images/
According to the above rule, all robots are denied access to the /images/ folder of a site. Additionally, a specific rule can be set to explicitly disallow access to all files within a folder:
Disallow:/images/*
/images/ and /images/* are the same thing. The path specified in a Disallow rule is simply a prefix. (Therefore a "wildcard" * is never required at the end of the path.)
– MrWhite
Sep 10 '14 at 13:07
No Allow code? Try add
User-agent: *
Allow: /
Disallow: /images/
Disallow: /upload/
Disallow: /admin/
