Linked Questions

3
votes
4answers
6k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
17
votes
3answers
10k views

Can robots.txt be in a server's sub-directory?

I have a sub-directory that I would like to hide from the search engine web crawlers. One way to do this is to use a robots.txt in the root directory of the server (the standard way). However, anyone ...
1
vote
3answers
160 views

URLs from my site appear in Google search results even though they are disallowed in robots.txt [duplicate]

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
3
votes
2answers
3k views

Why are new pages not being indexed and old pages stay in the index?

I currently have a site that was recently restructured, causing much of its content to be reposted, creating new URL's for each page. To avoid duplicates, all of the existing pages were added to the ...
2
votes
2answers
646 views

How to tell google that domain no longer exists but still make redirects from that domain

I have two domains: example.com and dev.example.net. example.com is the active one and dev.example.net should be invisible to Google. dev.example.net has 301 redirects to example.com and it has its ...
2
votes
2answers
5k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by robots....
1
vote
2answers
63 views

Impact of query parameters on SEO for a single page application

Might crawlers visit a page if there is no link referencing this page anywhere but a URL to this page is generated client side with JavaScript ? Context: Let's say I have a SPA with Server Side ...
1
vote
2answers
371 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
4
votes
1answer
3k views

How to make sure an individual page from a website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
2
votes
1answer
221 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
1
vote
1answer
15 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
1
vote
1answer
39 views

Google DMCA on a page that was forbidden in robots.txt

I've received a "Notice of DMCA removal from Google Search" for a page was inaccessible to google as per robots.txt. Why did Google have this page in search if it was disallowed from visiting it at ...
0
votes
0answers
14 views

When will search engines disregard rules in my robots.txt to NOT crawl my website and files?

I'm working with a marketer that wants to set up a website with a text field for people type in their guess of a url for another publicly accessible but secret website. Let's say that publicly ...
0
votes
0answers
17 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...

15 30 50 per page