Linked Questions

3
votes
4answers
6k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
3answers
150 views

URLs from my site appear in Google search results even though they are disallowed in robots.txt [duplicate]

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
16
votes
3answers
9k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
11
votes
5answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
8
votes
4answers
3k views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
9
votes
4answers
1k views

301 redirect to 404 page or set status code to 404 and stay on page?

I have a number of pages on my website that only administrators can access and access to these pages is given if a querystring value if found and correctly set. For example: http://www.mydomain.com/...
3
votes
2answers
3k views

Why are new pages not being indexed and old pages stay in the index?

I currently have a site that was recently restructured, causing much of its content to be reposted, creating new URL's for each page. To avoid duplicates, all of the existing pages were added to the ...
4
votes
5answers
480 views

With Robots.txt disallow all, why was my site still getting traffic?

In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because of a message in my AdSense account saying that five URL's couldn't be crawled. ...
4
votes
1answer
3k views

How to make sure an individual page from a website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
2answers
5k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by robots....
2
votes
2answers
573 views

How to tell google that domain no longer exists but still make redirects from that domain

I have two domains: example.com and dev.example.net. example.com is the active one and dev.example.net should be invisible to Google. dev.example.net has 301 redirects to example.com and it has its ...
1
vote
2answers
367 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
2
votes
1answer
215 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
2
votes
1answer
46 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...

15 30 50 per page