19 votes
Accepted

How do I find when an URL was first indexed by Google?

To know the age of an URL you can follow this link by replacing www.example.com by the URL you want: https://www.google.com/search?tbs=cdr%3A1%2Ccd_min%3A1%2F1%2F2000&q=site%3Ahttp%3A%2F%2Fwww....
  • 10k
13 votes

How does Google find a domain with no links to it?

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from ...
  • 32.7k
12 votes

How to make whole site headers send code 404?

This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go. There are many ...
  • 3,132
10 votes
Accepted

How do I prevent Google from indexing text that is generated by JavaScript?

robots.txt can block JavaScript files from Googlebot. http://www.robotstxt.org/ has more information about how to construct a robots.txt file. You could put your JavaScript that shows the password ...
  • 126
10 votes
Accepted

Replace all images when search engine crawler visits site

That's a very bad idea and your site will suffer in the organic search rankings. For one, Google does have image recognition abilities and so your assumption that the bot can't "view" the image is ...
  • 676
10 votes
Accepted

How to avoid robots from indexing pages of my app through alternate URLs?

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you ...
  • 5,957
10 votes
Accepted

How to allow Googlebot unlimited crawling but to have a limit on individual users?

Googlebot will be able to crawl your site fully because it never sends any cookies. There is, however, something else you need to do. Google expects sites to implement limits on the amount of content ...
10 votes
Accepted

Could a brand name called "Error 404" cause SEO issues?

You will be able to get your most of your content indexed, but there will probably be some SEO issues associated with that brand name. Google uses heuristics to determine if the page is an error page. ...
9 votes

How to make whole site headers send code 404?

If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers. robots....
  • 5,834
9 votes
Accepted

Is it worthwhile to pay for boosting SEO for a new wesite?

Never pay for SEO services that guarantee results. Especially not quick results. Anything quick is going to be dirty. Such firms are likely going to: Create links to your site from bad places ...
8 votes

How do I find when an URL was first indexed by Google?

Zistoloen found a way to have Google display the date when it first indexed the content of the page. I'm adding it to my answer as well because I think I can explain it more clearly. Search Google ...
7 votes
Accepted

Number of indexed pages with "site:" search less than reported in Google Webmaster Tools

You do not have an indexing problem. Google Webmasters Tools is the authoritative source of information about your website with Google. Whatever it says there is true. Operators like site: and link: ...
  • 85.9k
7 votes
Accepted

Why doesn't searching for the unique name of my website bring the site up in a top position on Google?

No SEO campaign is perfect. So please throw this notion out of the window. Since search engines change strategies and updates come on the average of several a day, it is impossible to have a perfect ...
  • 32.7k
7 votes
Accepted

Will Google penalize me for duplicate content on GitHub from my knowledge base?

Ideally, no. It is very, very difficult to get penalised (Sandboxing, Deindexing). Having duplicate content may 'de-value' your content and it will have less weight than it would organically but it's ...
  • 1,128
7 votes
Accepted

Is it bad for SEO to submit a partially finished site to Google?

There is no problem with showing Google a small site that you plan to add to later. If you have one page of good content, Google would like to index it because it could be useful to somebody ...
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with & ... ...
  • 42k
7 votes
Accepted

How to tell bots to forget a website and reindex it from scratch

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
6 votes

How to get tens of millions of pages indexed by Google bot?

I've had experience of just this sort of site. I ran an article directory many years ago and the % of pages indexed and more importantly actually performing was pretty much directly correlated to the ...
6 votes

Number of indexed pages with "site:" search less than reported in Google Webmaster Tools

You had posted this question differently twice. This answer was to the question marked as duplicate which I now post here hoping to help you. While the two questions may not be exactly the same, this ...
  • 32.7k
6 votes

Huge drop off in Google "indexed" pages after switching to HTTPS, should I switch back to HTTP?

We just switched from HTTP to HTTPS two weeks ago, and after doing extensive research this is what we did: Added the HTTPS property to our Search console. Redirected HTTP robots.txt to HTTPS robots....
  • 679
6 votes
Accepted

Is it possible for Google to disregard my canonical tag?

Early 2018 update to Google Search Console will now tell you if it's overridden a page with a different canonical: Google chose different canonical than user: This URL is marked as canonical for a ...
  • 3,132
6 votes
Accepted

Do SEO analysis bots have any advantage in terms of website traffic?

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
5 votes
Accepted

Is it bad to have both www and the naked domain indexed in Google? Should both be added to webmaster tools?

I would recommended either using the www or non-www and stick with it. Set a preferred URL in Webmaster Tools and then 301 to either your www or non-www. The reason for this would be Google would ...
5 votes
Accepted

Change to www from naked domain impact on seo

Will you lose SEO value? 301-headers will transfer all* SEO value from the naked version to the www version, you have nothing to worry about, as long as the pages keep existing (which should be the ...
  • 6,776
5 votes

Number of indexed pages with "site:" search less than reported in Google Webmaster Tools

It could also be duplicate content filters. Do a site search then go to last page of search results. This may be page 30 or more. At the bottom of the last page you should see something similar to ...
  • 3,319
5 votes
Accepted

Prevent Googlebot from seeing or indexing common keywords such as "read full story"

I think you answered your own question in the first statement. Using the same anchor text for a lot of your links is not a good idea. Instead, try using something more descriptive. However, this is ...
  • 3,658
5 votes

Best way to let Google index page with tabs - SEO

This is a very common problem faced these days. This can be easily solved as follows: If you want to have same page /pets but also drive /pets/cats, /pets/dog to index separately, then ideally they ...
  • 1,694
5 votes
Accepted

Redirecting robots.txt when moving to a new domain

This is really interesting question, It took me few minutes to search official statement from gary tweet.. When Google see 404/30X status, then Google will ignore your robots.txt completely. It is ...
  • 5,497
5 votes

Why Google is indexing Thumbnails and not Large images?

The issue is that Google only sees one ALT tag therefore the other image is without description and will not rank in image search results. If using a lightbox then preferably you should code or use ...
  • 32.7k
5 votes
Accepted

How can I remove this URL from Google?

It looks like that you have affiliate links which have 302 redirected to the Amazon or another affiliate site. This is a common issue for affiliate websites. The issue with 302 redirects is that ...
  • 544

Only top scored, non community-wiki answers of a minimum length are eligible