Linked Questions

2
votes
1answer
19 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
18
votes
3answers
10k views

Are web crawlers able to find a secondary robots.txt in a sub-directory?

I have a sub-directory that I would like to hide from web crawlers. One way to do this is to use a robots.txt in the root directory of the server and add a rule to exclude that sub-directory. However,...
0
votes
0answers
14 views

When will search engines disregard rules in my robots.txt to NOT crawl my website and files?

I'm working with a marketer that wants to set up a website with a text field for people type in their guess of a url for another publicly accessible but secret website. Let's say that publicly ...
1
vote
2answers
67 views

Impact of query parameters on SEO for a single page application

Might crawlers visit a page if there is no link referencing this page anywhere but a URL to this page is generated client side with JavaScript ? Context: Let's say I have a SPA with Server Side ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
45 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
2
votes
2answers
5k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by robots....
1
vote
3answers
162 views

URLs from my site appear in Google search results even though they are disallowed in robots.txt [duplicate]

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
11
votes
5answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
9
votes
4answers
2k views

301 redirect to 404 page or set status code to 404 and stay on page?

I have a number of pages on my website that only administrators can access and access to these pages is given if a querystring value if found and correctly set. For example: http://www.mydomain.com/...
2
votes
2answers
688 views

How to tell google that domain no longer exists but still make redirects from that domain

I have two domains: example.com and dev.example.net. example.com is the active one and dev.example.net should be invisible to Google. dev.example.net has 301 redirects to example.com and it has its ...
4
votes
1answer
3k views

How to make sure an individual page from a website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
4
votes
4answers
7k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
373 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
4
votes
5answers
482 views

With Robots.txt disallow all, why was my site still getting traffic?

In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because of a message in my AdSense account saying that five URL's couldn't be crawled. ...
8
votes
4answers
3k views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
3
votes
2answers
3k views

Why are new pages not being indexed and old pages stay in the index?

I currently have a site that was recently restructured, causing much of its content to be reposted, creating new URL's for each page. To avoid duplicates, all of the existing pages were added to the ...
2
votes
1answer
221 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...