Linked Questions

0
votes
0answers
15 views

When will search engines disregard rules in my robots.txt to NOT crawl my website and files?

I'm working with a marketer that wants to set up a website with a text field for people type in their guess of a url for another publicly accessible but secret website. Let's say that publicly ...
2
votes
1answer
66 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
1
vote
2answers
79 views

Impact of query parameters on SEO for a single page application

Might crawlers visit a page if there is no link referencing this page anywhere but a URL to this page is generated client side with JavaScript ? Context: Let's say I have a SPA with Server Side ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
47 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
2
votes
2answers
733 views

How to tell google that domain no longer exists but still make redirects from that domain

I have two domains: example.com and dev.example.net. example.com is the active one and dev.example.net should be invisible to Google. dev.example.net has 301 redirects to example.com and it has its ...
18
votes
3answers
10k views

Are web crawlers able to find a secondary robots.txt in a sub-directory?

I have a sub-directory that I would like to hide from web crawlers. One way to do this is to use a robots.txt in the root directory of the server and add a rule to exclude that sub-directory. However,...
1
vote
3answers
163 views

URLs from my site appear in Google search results even though they are disallowed in robots.txt [duplicate]

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
9
votes
4answers
2k views

301 redirect to 404 page or set status code to 404 and stay on page?

I have a number of pages on my website that only administrators can access and access to these pages is given if a querystring value if found and correctly set. For example: http://www.mydomain.com/...
1
vote
2answers
373 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
4
votes
4answers
7k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
2answers
6k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by robots....
4
votes
5answers
482 views

With Robots.txt disallow all, why was my site still getting traffic?

In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because of a message in my AdSense account saying that five URL's couldn't be crawled. ...
8
votes
4answers
3k views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...

15 30 50 per page