Google doesn't seem to crawl the homepage of my site

Question

I am rather new to the webmaster world but have run into a problem I can't seem to find a solution to. If I search for my site suleimanholdings.com the first results from my site are secondary pages instead of the homepage. I also tried to find the homepage by using "site:suleimanholdings.com" and was given about 7 results (the site is very small right now so that should be a low number) but the homepage isn't one of them.

The site is built using Meteor which does complicate crawling a bit however I installed a spiderable package that should take care of it, and it does seem to work as every page except the homepage will come up on Google for me.

I tried running a test through online curl to check the HTTP status and was given a 200 OK. Here are the rest of those results:

HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Date: Wed, 12 Aug 2015 18:07:02 GMT
Connection: keep-alive
Transfer-Encoding: chunked

In addition I do have both the www and non-www options on my Google Webmaster Tools. I have created and submitted sitemaps and even individually fetched and rendered pages, then submitted them to the index. Strangely enough although the pages were submitted to the index and the sitemap was submitted, if I visit the section to see what pages or how many have been indexed it tells me that none at all have been indexed.

Why would just the homepage of site not be indexed by Google?

While the question was not quite the same, the answer has some nuggets of information that should help you understand a few things. We have answered these questions and I know there is a better answer out there- possibly one of mine. But since this one was handy, I thought I would share: webmasters.stackexchange.com/questions/83908/… — closetnoc, Aug 12, 2015 at 19:06
Thanks for including that link. I read through your answer but am not completely satisfied. I understand why it wouldn't come up first thing in a general search, or even using the site: search option, however my problem is that it doesn't come up at all. It's a very small site, only around 7 or 8 pages and even expanding the site: search option, it's nowhere to be found? I feel as though there might be some crawl error but am not able to locate it as it doesn't list anything for me. — Kristen Vogler, Aug 12, 2015 at 19:14
It's possible that it's because your homepage has literally ZERO content without the Javascript being executed. Though the other pages are the same, so not sure that'd be the issue. — nathangiesbrecht, Aug 12, 2015 at 19:47
@nathangiesbrecht I agree I thought perhaps that could be the problem (It is built through meteor which comes down to node.js), but with the spiderable package installed it takes care of that by telling google to add "?_escaped_fragment_=" and that basically generates an html version for google to read. — Kristen Vogler, Aug 12, 2015 at 19:51

heytools · Accepted Answer · 2015-08-12 19:00:24Z

2

You may want to give it some more time, I already see 10 results and the home page is the first result:

answered Aug 12, 2015 at 19:00

heytools

3861 silver badge11 bronze badges

1

Thanks for sharing that screenshot. Very strange that I cannot find a similar screen even though I have tried three different browsers. Another note is, that listing is an old version from 2-3 months ago, I can tell by the title as it doesn't follow the updated version with | Home. So even though it appears listed for you it hasn't actually been updated in a few months, although the other listings clearly have
– Kristen Vogler
Aug 12, 2015 at 19:11
strange enough, the cached version is about the same as the current website and its title is the new "Home | Suleiman Holdings" - webcache.googleusercontent.com/…
– heytools
Aug 12, 2015 at 19:24
@KristenVogler Google doesn't necessarily use the title of your page as the title in the SERPs, so this isn't necessarily an indication that the index has been updated.
– MrWhite
Aug 13, 2015 at 0:05

Add a comment |

closetnoc · Accepted Answer · 2015-08-12 20:21:02Z

Actually this is quite an easy one.

Your home page has no content. There is nothing for Google to chew on except the title tag. That is not enough. While the home page looks fantastic(!) (and it does by the way), search engines require content- at least enough to create a SERP link. In this case, you have not supplied the minimum for a SERP link.

You can take this in incremental stages. I suggest adding a description meta-tag at least to see if Google will at least create a SERP link for you. You can use the Google Search Console (application formerly know as Webmaster Tools [80's flashback anyone?]) and use Fetch as Google to refetch the page. This will happen right away, but may not hit the index for a while. Google will always revisit after the fetch with the Googlebot crawler a few hours or days and then index your home page properly.

If this is not enough, then I suggest some minimal content- an h1 tag and a paragraph. I rather suspect that just adding the description meta-tag should be enough. You may have to wait a while.

But know this. Because there is no content, the home page may never really rank for anything. Part of this is because your title tag is rather nondescript. I suggest at least making the title tag and description meta-tag the best they can be.

view-source:http://suleimanholdings.com/

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" class="__meteor-css__" href="/d4c7cfcf445710b33c839278e1fcd641da7e874e.css?meteor_css_resource=true">


<script type="text/inject-data">%7B%22fast-render-data%22%3A%7B%22collectionData%22%3A%7B%22users%22%3A%5B%5B%7B%22_id%22%3A%22vpGcNX9zap9nXSo7H%22%7D%5D%5D%7D%2C%22subscriptions%22%3A%7B%7D%7D%7D</script>
<script type="text/javascript">__meteor_runtime_config__ = JSON.parse(decodeURIComponent("%7B%22meteorRelease%22%3A%22METEOR%401.1.0.3%22%2C%22PUBLIC_SETTINGS%22%3A%7B%22ga%22%3A%7B%22id%22%3A%22UA-62610011-1%22%7D%7D%2C%22ROOT_URL%22%3A%22http%3A%2F%2Fsuleimanholdings.com%22%2C%22ROOT_URL_PATH_PREFIX%22%3A%22%22%2C%22autoupdateVersion%22%3A%22aa4d4d0a62689c78dcf3e156a73fb3eb8d556344%22%2C%22autoupdateVersionRefreshable%22%3A%2213364f1b58a9e01cfdbb742f20d5385cb4847e1b%22%2C%22autoupdateVersionCordova%22%3A%22none%22%7D"));</script>

  <script type="text/javascript" src="/47cdc2acdb58961b1e38a332a698031122ac41bb.js"></script>



<script id="irga-analytics" async src='//www.google-analytics.com/analytics.js'></script>
<meta name="fragment" content="!">
<meta charset="UTF-8">
    <title>Suleiman Holdings</title>
    <link rel="icon" href="/favicon.ico?v=2">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="css/style.css" rel="stylesheet" type="text/css">
    <link rel="stylesheet" href="css/reset.css">
    <link rel="stylesheet" href="css/flexslider.css" type="text/css" media="screen" />
    <script src="/prefixfree.min.js"></script>
    <script src="/jquery.flexslider.js"></script>
    <script>
        (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
                    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
                m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
        })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

        ga('create', 'UA-62610011-1', 'auto');
        ga('send', 'pageview');

    </script>
</head>
<body>

</body>
</html>

view-source:http://suleimanholdings.com/about

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" class="__meteor-css__" href="/d4c7cfcf445710b33c839278e1fcd641da7e874e.css?meteor_css_resource=true">


<script type="text/inject-data">%7B%22fast-render-data%22%3A%7B%22collectionData%22%3A%7B%22users%22%3A%5B%5B%7B%22_id%22%3A%22vpGcNX9zap9nXSo7H%22%7D%5D%5D%7D%2C%22subscriptions%22%3A%7B%7D%7D%7D</script>
<script type="text/javascript">__meteor_runtime_config__ = JSON.parse(decodeURIComponent("%7B%22meteorRelease%22%3A%22METEOR%401.1.0.3%22%2C%22PUBLIC_SETTINGS%22%3A%7B%22ga%22%3A%7B%22id%22%3A%22UA-62610011-1%22%7D%7D%2C%22ROOT_URL%22%3A%22http%3A%2F%2Fsuleimanholdings.com%22%2C%22ROOT_URL_PATH_PREFIX%22%3A%22%22%2C%22autoupdateVersion%22%3A%22aa4d4d0a62689c78dcf3e156a73fb3eb8d556344%22%2C%22autoupdateVersionRefreshable%22%3A%2213364f1b58a9e01cfdbb742f20d5385cb4847e1b%22%2C%22autoupdateVersionCordova%22%3A%22none%22%7D"));</script>

  <script type="text/javascript" src="/47cdc2acdb58961b1e38a332a698031122ac41bb.js"></script>



<script id="irga-analytics" async src='//www.google-analytics.com/analytics.js'></script>
<meta name="fragment" content="!">
<meta charset="UTF-8">
    <title>Suleiman Holdings</title>
    <link rel="icon" href="/favicon.ico?v=2">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="css/style.css" rel="stylesheet" type="text/css">
    <link rel="stylesheet" href="css/reset.css">
    <link rel="stylesheet" href="css/flexslider.css" type="text/css" media="screen" />
    <script src="/prefixfree.min.js"></script>
    <script src="/jquery.flexslider.js"></script>
    <script>
        (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
                    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
                m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
        })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

        ga('create', 'UA-62610011-1', 'auto');
        ga('send', 'pageview');

    </script>
</head>
<body>

</body>
</html>

This is what I see when I visit your site and do a view source for what it is worth. When you are done, I, or anyone, can edit these out for you.

Comments are not for extended discussion; this conversation has been moved to chat. — John Conde, Aug 12, 2015 at 23:53
the screenshot from @heytools shows that it does display results and a snippet underneath — Mousey, Aug 13, 2015 at 0:22
@Mousey We already determined that was an older listing- probably hitting another data-center. Thanks!! — closetnoc, Aug 13, 2015 at 0:33
@closetnoc the home page has the fragment meta tag and an ugly URL with the HTML snapshot has been created for the home page. — Mousey, Aug 13, 2015 at 11:05

Mike -- No longer here · Accepted Answer · 2015-08-12 20:20:55Z

0

I took a look at the visual and the source code of the page and agree that the home page has no content.

I also ran an additional test to run the page with javascript disabled and nothing appears on screen.

You may want to run your page though a bunch of SEO testing tools. Also, try having googlebot test your site in Webmaster tools, and if possible, download a text-only web browser and test your pages in that.

You want actual content on screen without the requirement of an external script or external file to load the content. It makes loading time much smaller that way.

answered Aug 12, 2015 at 20:20

Mike -- No longer here

13.6k4 gold badges27 silver badges58 bronze badges

Thanks Mike, as I was trying to explain before every page of this site reveals no content when using the page source option. I am not fighting that at all, my question is this, why will google index correctly the other pages which all have the same issue but it will not correctly do it to the homepage?
– Kristen Vogler
Aug 12, 2015 at 20:33
You answered your own question in your comment. Google does not like indexing pages that its robot believe to have no content, and a bunch of HTML tags do not count as content. Also in many search engines, javascript code does not count as content either.
– Mike -- No longer here
Aug 12, 2015 at 21:35
1

As I said before Mike, every page on the site is built the exact same, so if you view any of them with page source it will show the missing content, however the homepage seems to be the only one having the indexing issue, the other pages are indexed even without content. By content I am referring to the text and images inside the html tags.
– Kristen Vogler
Aug 12, 2015 at 21:38
Do you have a URL that you can show that you say that has no content that google is indexing?
– Mike -- No longer here
Aug 12, 2015 at 21:39
1

Then what I would recommend is make googlebot refetch your homepage and submit it to the index again. Sometimes google is slow at updating pages.
– Mike -- No longer here
Aug 12, 2015 at 22:02

| Show 4 more comments

Mousey · Accepted Answer · 2015-08-13 16:32:00Z

This solution involves giving the crawler an HTML snapshot for dynamically created content.

For pages without fragments - see 3. in the link - these are handled using meta tags to indicate the dynamic content Once HTML snapshots exist the dynamic page will direct the googlebot and other crawlers there.

The sitemap should also give the HTML snapshots rather than the dynamic pages, allowing non-google search engines to crawl it too.

Once the Fetch as Google sees the HTML snapshot the Data Highlighter can be used as a quick way to highlight any structured data. This should also help SEO.

Edit Websniffer results Websniffer (an alternative to Fetch as google) shows a script is running before the fragment meta tag. Move the meta tag higher, then do Fetch as google.

Secondly, <meta name="fragment" content="!"> should not be in the HTML snapshot/ugle URL page since this actually tells the crawler to go elsewhere.

Once Fetch as google succeeds in returning the visible content it can take a few days at least before the search results are updated.

Edit 2 There appears to be a bug in Google webmaster tools which causes it to appear as if it isn't being crawled, even when the crawling and indexing do work, which is what seems to be the case here.

meteor SEO without spiderable is also possible

Setting the ugly URL for the HTML snapshot - looks correct

The instructions state that:

If a page has no hash fragments, but contains in the of the HTML, the crawler will transform the URL of this page fromdomain[:port]/path to domain[:port]/path?_escaped_fragment= (ordomain[:port]/path?queryparams to domain[:port]/path?queryparams&_escaped_fragment_= and will then access the transformed URL. For example, if www.example.com contains in the head, the crawler will transform this URL into www.example.com?_escaped_fragment_= and fetch www.example.com?_escaped_fragment_= from the web server.

so the .htaccess or similar will need to be set to transform the URL http://suleimanholdings.com/path?escaped_fragment_= to automatically reach the ugly HTML. The home page ugly URL is http://suleimanholdings.com?_escaped_fragment_= which works.

I believe I have done that. That listed meta tag is included. I could add even load the ugly urls and they appear correctly. But the page source still displays no body content? — Kristen Vogler, Aug 13, 2015 at 1:03
@KristenVogler I've edited the answer significantly. Apologies for missing the fragment meta tag and HTML snapshot first time around. — Mousey, Aug 13, 2015 at 11:03
Thanks for the update, The site is built using meteor so to make it crawlable it uses a package called spiderable that uses phantomJS as well. So that meta tag gets placed automatically and I can't move it. When I visit the ugly url it seems to do a weird duplicating thing where it shows me the head content a bunch of times before my html code, which eventually does appear at the end. I am wondering if there is perhaps some other error between the packages that is causing the weird duplication and perhaps I can fix that to solve the overall problem — Kristen Vogler, Aug 13, 2015 at 14:18
@Kristen Vogler - see Edit 2. It certainly does seem to be crawling your home page based on search results. If the crawl is out of date can you give a clear example of what should display instead? — Mousey, Aug 13, 2015 at 16:34
my guess is you are seeing what Mike and heytools saw, the homepage come up with Suleiman Holdings : Home as the title when 3 months ago it was changed to Home | Suleiman Holdings, so I am relatively certain google isn't able to update that page for some reason. — Kristen Vogler, Aug 13, 2015 at 17:34

Stack Exchange Network

Google doesn't seem to crawl the homepage of my site

4 Answers 4

Your Answer

Not the answer you're looking for? Browse other questions tagged
google-search-console
google-index
or ask your own question.

Linked

Hot Network Questions

Google doesn't seem to crawl the homepage of my site

4 Answers 4

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged google-search-consolegoogle-index or ask your own question.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
google-search-console
google-index
or ask your own question.