Linked Questions

0
votes
0answers
14 views

When will search engines disregard rules in my robots.txt to NOT crawl my website and files?

I'm working with a marketer that wants to set up a website with a text field for people type in their guess of a url for another publicly accessible but secret website. Let's say that publicly ...
1
vote
1answer
15 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
1
vote
2answers
63 views

Impact of query parameters on SEO for a single page application

Might crawlers visit a page if there is no link referencing this page anywhere but a URL to this page is generated client side with JavaScript ? Context: Let's say I have a SPA with Server Side ...
17
votes
3answers
10k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
39 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
2
votes
2answers
5k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by robots....
1
vote
3answers
160 views

URLs from my site appear in Google search results even though they are disallowed in robots.txt [duplicate]

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
11
votes
5answers
16k views

Can I invoke Google to check my robots.txt?

I read the answers in this question, but they still leave my question open: Does Google cache robots.txt? I didn't find a way in the Google Webmaster Tools to invoke a re-download of my robots.txt. ...
9
votes
4answers
2k views

301 redirect to 404 page or set status code to 404 and stay on page?

I have a number of pages on my website that only administrators can access and access to these pages is given if a querystring value if found and correctly set. For example: http://www.mydomain.com/...
2
votes
2answers
646 views

How to tell google that domain no longer exists but still make redirects from that domain

I have two domains: example.com and dev.example.net. example.com is the active one and dev.example.net should be invisible to Google. dev.example.net has 301 redirects to example.com and it has its ...
4
votes
1answer
3k views

How to make sure an individual page from a website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
3
votes
4answers
6k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
1
vote
2answers
371 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...

15 30 50 per page