33

This is a general, community wiki catch-all question and answer pair intended to address any questions concerning the reasons a site or specific site contents do not appear in search engine results.

If your question was closed as a duplicate of this question and you feel that the information provided here does not provide a sufficient answer, please open a discussion on Pro Webmasters Meta.

My site (or specific pages on my site) is not appearing in search engine results.

Why isn't my content indexed and what can I do about it?

3 Answers 3

41

There are a number of reasons your content may not appear in search engine results, however, it is important to note that a search engine's index may contain pages that it doesn't display in its results page.

How to tell if your content is actually indexed

It may actually be difficult to tell if your content is indexed.

  • Search for all the documents from your site and see how many are listed

    • Google: enter site:example.com (where example.com is your domain, there must not be any space after the colon.)
    • Bing: enter site: example.com
    • Yahoo: enter site: example.com (or use advanced search form)
  • Search for a specific document by a unique sentence of eight to twelve words and search for that sentence in quotes. For example, to find this document, you might choose to search for "number of reasons your content may not appear in search engine results"

  • In addition to above, search for keywords using inurl: and intitle: you may try something like, keyword with another keyword inurl:example.com this will bring upi pages that are indexed only for specified domain.

  • Log into webmaster tools to see stats from the search engine itself about how many pages are indexed from the site

    • Google Webmaster Tools – Information is available under "Health" » "Index Status". If you have submitted site maps, you can also see how many documents in each site map file have been indexed.
    • Bing Webmaster Tools

In some cases, documents may not appear to be indexed via one of these methods, but documents can be found in the index using other methods. For example, webmaster tools may report that few documents are indexed even when you can search for their sentences and find the documents on the search engine. In such a case, the documents are actually indexed.

How content becomes indexed

Before search engines index content, they must find it using a web crawler. You should check your webserver's logs to see whether search engines' crawlers (identified by their user agent – e.g. Googlebot, Bing/MSNbot) are visiting your site.

Larger search engines like Google and Bing typically crawl sites frequently, but the crawler may not know about new site. You can notify search engines to the existence of your site by registering as its webmaster (Google Webmaster Tools, Bing Webmaster Tools) or, if the search engine does not provide this facility, submitting a link to its crawlers (e.g. Yahoo).

How long has your site/content been online?

Search engines may index content very quickly after it has been found, however, these updates are occasionally delayed. Smaller search engines can also be much less responsive and take weeks to index new content.

If your site hasn't been live for more than a few months, the search engines may not trust it enough to index much content from it yet.

Do other websites link to your content?

If your content has only been online for several days and does not have any links from other sites (or its links come from sites which crawlers do not visit frequently) it is probably not indexed.

You may be able to speed up indexing through white-hat techniques for attracting high quality inbound links, such as by linking to your content (when relevant) from your related social media account or blog, and by creating content that is compelling enough that other websites naturally want to link to it.

Has the content been excluded by the webmaster?

This step is especially important if you are taking over a site from someone else and there is an issue with a specific page or directory: check for robots.txt and META robots exclusions and remove them if you want crawlers to index the content being excluded.

Is there a technical issue preventing your content from being indexed?

If you have an established site but specific content is not being indexed (there are no web crawler hits on the URLs where the content resides) the webmaster tools provided by Google and Bing may provide useful diagnostic information.

Google's Crawl Errors documentation provides extensive background on common problems for web crawlers which prevent content from being indexed and, if you use Google Webmaster Tools, you will receive an alert if any of these issues are detected on your site.

Correct errors and misconfigurations as quickly as possible to ensure that all of your site's content is indexed.

Is the content low quality?

Search engines don't index most pages they crawl. They only index the highest quality content. Search engines will not index content if:

  • It is spam, gibberish, or nonsense.
  • It is found elsewhere. When search engines find duplicate content, they choose only one of the duplicates to index. Usually that is the original that has more reputation and links.
  • It is thin. It needs more than a couple lines of original text. Preferably much more. Automatically created pages with little content such as a page for each of your users are unlikely to get indexed.
  • It doesn't have enough reputation or links. A page may be buried too deep in your site to rank. Any page without external links and more than a few clicks from the home page is unlikely to get indexed.

Is some of your content indexed, but not all?

If your site has hundreds of pages, Google will almost never choose to index every single page. If you site has tens of thousands of pages, it is very common for Google to choose to index only a small portion of those pages.

Google chooses the number of pages to index from a site based on the site's overall reputation and the quality of the content. Google typically indexes a larger percent of a site over time as the site's reputation grows.

9
  • For my site mfaani.com I'm seeing Google say 38 results, but when I go the next page, it turns into only 16 results. Is that expected? Is there a way around that?
    – Honey
    Mar 25, 2022 at 19:15
  • All interesting points, but basically, Google is ignoring most of all sites. Even for five SEO experts's sites I checked for how many pages were searchable compared to those listed in their sitemap.xml file(s), the findability was only 28%. Whatever excuses are used to try to explain why there is no full coverage, it still comes down to Google just plain ignoring most pages. That is something that all these legislators wanting to go after Google should know. Google is misrepresenting themselves because any reasonable person expects a global search giant to make all public pages findable.
    – Patanjali
    Apr 22, 2022 at 23:52
  • Basically, lacked of findability means that if someone comes across one of your site's pages that is not listed by Google, if that person then tries to find that page again by searching Google later, they will not find it.
    – Patanjali
    Apr 22, 2022 at 23:57
  • 1
    With my own personal site with over 150 pages of original content, most of which has been around for years, Google only ever shows 12 (8%) of them in site:-prefixed searches. At least Bing shows 39% of them. For less popular content, Bing might be better for searching, though still falls short
    – Patanjali
    Apr 23, 2022 at 0:07
3

I think I don't need to repeat the same thing as stated above, however if you're not familiar with new tool then here is the way to check it out below.

You can check if your robot.txt file is not blocking the content by going to old Google Webmaster's robots.txt Tester Tool

enter image description here

Google Search Console (new version)

Google search console provides valuable information than older version. Check for the issues right from Google Search Console, whether your page is crawled or not, indexed or not, live or not and the issues if any.

  1. Log in to your website's Google Search Console
  2. If you've multiple domain properties then make sure you've selected the desired one.

Deprecated Tool in Google Webmaster

This is outdated now

This is how you can go through the new Search Console

New Google Search Console

1
  • is there a way to automate this? for example feed it with a csv file and it request indexing?
    – Rain Man
    Jul 26, 2023 at 18:27
0

If you have a robots.txt file in your root directory, it might be telling Google not to index your site.

Have you checked for that?

Usually it is safe to just delete (or rename) those files since it sounds like you DO want to be indexed.

IMPORTANT: This can be a deal-breaker if you are trying to get Google Adsense to work on your site.

No matter where you put your special code, if you are not being indexed you will be told your site is down so they can not verify it.

1
  • robots.txt file has user-defined set of rules to crawl or not, so removing it is not a solution as this is the first file to be crawled. Best solution is fix the rules, test robots.txt file add sitemap directive in your robot.txt keep your sitemap up to date with lastmod and change frequency attributes. Apr 13, 2019 at 13:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.