19 votes
Accepted

How do I find when an URL was first indexed by Google?

To know the age of an URL you can follow this link by replacing www.example.com by the URL you want: https://www.google.com/search?tbs=cdr%3A1%2Ccd_min%3A1%2F1%2F2000&q=site%3Ahttp%3A%2F%2Fwww....
user avatar
  • 10k
17 votes
Accepted

Google is still crawling and indexing my old, dummy, test pages which now are 404 not found

Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is ...
user avatar
  • 32.5k
13 votes

How does Google find a domain with no links to it?

There are at least 3 ways: Links to your site. Using Google Webmaster Tools (now called Search Console) Registrar dumps, triggers, and other options. Google will find many new sites quickly from ...
user avatar
  • 32.5k
12 votes

How to make whole site headers send code 404?

This is an XY problem. You want to prevent indexing on your site and you know that 404s are not indexed, so you want to prevent indexing 'using' 404s. This is the wrong way to go. There are many ...
user avatar
  • 2,727
10 votes
Accepted

How do I prevent Google from indexing text that is generated by JavaScript?

robots.txt can block JavaScript files from Googlebot. http://www.robotstxt.org/ has more information about how to construct a robots.txt file. You could put your JavaScript that shows the password ...
user avatar
  • 126
10 votes
Accepted

Replace all images when search engine crawler visits site

That's a very bad idea and your site will suffer in the organic search rankings. For one, Google does have image recognition abilities and so your assumption that the bot can't "view" the image is ...
user avatar
  • 676
10 votes
Accepted

How to avoid robots from indexing pages of my app through alternate URLs?

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you ...
user avatar
  • 5,887
10 votes
Accepted

How to allow Googlebot unlimited crawling but to have a limit on individual users?

Googlebot will be able to crawl your site fully because it never sends any cookies. There is, however, something else you need to do. Google expects sites to implement limits on the amount of content ...
user avatar
10 votes

Could a brand name called "Error 404" cause SEO issues?

You will be able to get your most of your content indexed, but there will probably be some SEO issues associated with that brand name. Google uses heuristics to determine if the page is an error page. ...
user avatar
9 votes
Accepted

Is there a way to bulk submit URLs to Fetch as Google?

Indirectly, yes: Create a page with links to all the URLs you'd like re-crawled (like a sitemap) and add that to your website.* Submit the URL to that page to Fetch as Google, selecting Desktop as ...
user avatar
  • 15k
9 votes

How to make whole site headers send code 404?

If this is a test site that shouldn't be indexed at all, there are a couple of steps you can take that tell search engines not to index your site more effectively than returning 404 headers. robots....
user avatar
  • 5,794
9 votes
Accepted

Is it worthwhile to pay for boosting SEO for a new wesite?

Never pay for SEO services that guarantee results. Especially not quick results. Anything quick is going to be dirty. Such firms are likely going to: Create links to your site from bad places ...
user avatar
8 votes

Google is still crawling and indexing my old, dummy, test pages which now are 404 not found

Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally. To prevent the ...
user avatar
8 votes

What is Google's policy about separate content on same URLs in mobile and desktop versions?

The best course of action is to use canonical URLs. This avoids a situation where you are penalized for duplicate content. When it comes to desktop vs mobile websites, most sites will have something ...
user avatar
  • 2,299
8 votes

How do I find when an URL was first indexed by Google?

Zistoloen found a way to have Google display the date when it first indexed the content of the page. I'm adding it to my answer as well because I think I can explain it more clearly. Search Google ...
user avatar
7 votes

What is Google's policy about separate content on same URLs in mobile and desktop versions?

In my experience, mobile visitors want the same content as your desktop visitors do. I worked for a travel website with lots of information about hotels and restaurants. The site is generally known ...
user avatar
7 votes
Accepted

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the ...
user avatar
  • 1,424
7 votes
Accepted

Number of indexed pages with "site:" search less than reported in Google Webmaster Tools

You do not have an indexing problem. Google Webmasters Tools is the authoritative source of information about your website with Google. Whatever it says there is true. Operators like site: and link: ...
user avatar
  • 85.8k
7 votes
Accepted

Why doesn't searching for the unique name of my website bring the site up in a top position on Google?

No SEO campaign is perfect. So please throw this notion out of the window. Since search engines change strategies and updates come on the average of several a day, it is impossible to have a perfect ...
user avatar
  • 32.5k
7 votes
Accepted

Will Google penalize me for duplicate content on GitHub from my knowledge base?

Ideally, no. It is very, very difficult to get penalised (Sandboxing, Deindexing). Having duplicate content may 'de-value' your content and it will have less weight than it would organically but it's ...
user avatar
  • 1,108
7 votes
Accepted

Is it bad for SEO to submit a partially finished site to Google?

There is no problem with showing Google a small site that you plan to add to later. If you have one page of good content, Google would like to index it because it could be useful to somebody ...
user avatar
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with & ... ...
user avatar
  • 41.3k
7 votes
Accepted

How to tell bots to forget a website and reindex it from scratch

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
user avatar
6 votes

How can I fix Google indexing URLs that are not part of my sitemap?

Most likely some part of your web site generated links like that, and that is how Google started to crawl the URLs. You should check the links in your web pages to see where these incorrect URLs are, ...
user avatar
6 votes

Number of indexed pages with "site:" search less than reported in Google Webmaster Tools

You had posted this question differently twice. This answer was to the question marked as duplicate which I now post here hoping to help you. While the two questions may not be exactly the same, this ...
user avatar
  • 32.5k
6 votes

Huge drop off in Google "indexed" pages after switching to HTTPS, should I switch back to HTTP?

We just switched from HTTP to HTTPS two weeks ago, and after doing extensive research this is what we did: Added the HTTPS property to our Search console. Redirected HTTP robots.txt to HTTPS robots....
user avatar
  • 669
6 votes
Accepted

Is it possible for Google to disregard my canonical tag?

Early 2018 update to Google Search Console will now tell you if it's overridden a page with a different canonical: Google chose different canonical than user: This URL is marked as canonical for a ...
user avatar
  • 2,727
6 votes
Accepted

Do SEO analysis bots have any advantage in terms of website traffic?

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
user avatar
5 votes
Accepted

Request Removal of naked domain from Google Index

Google Webmaster Tools has a setting for it. Sign in to Webmaster Tools Add, verify, and select your website Use the gear icon and select "Site Settings" From "Preferred Domain", select "Display URLs ...
user avatar
5 votes

Google is still crawling and indexing my old, dummy, test pages which now are 404 not found

Google is likely to continue trying to crawl these pages for a long time. Webmasters make mistakes, or sites become unavailable for whatever reason, so Google won't remove content at the first sign of ...
user avatar
  • 41.3k

Only top scored, non community-wiki answers of a minimum length are eligible