2

I've looked over various threads here but nothing that seems to be the same issue. We're currently facing two problems at the moment. If I can address the first problem then it would definitely help diagnose the second issue.

Also, I've posted this to google's webmaster forum but no replies yet.

Our stack:

AngularJS, HTML + SCSS, AmazonS3 as our "web server" but as you may know is not really a web server. We have a redirect rule on the bucket to prefix any URL with a hash bang so the site functions properly. CloudFront in front of our S3 bucket.

First problem:

The "Fetch as Google" tool is truncating any url starting with a #! (hash bang) making it difficult to know if any of these pages can be crawled by Google. If this is working for other sites then the problem might be that we're using AmazonS3 as our "web server." I've checked other threads here and it seems to be working for other people.

Second problem:

Google is only indexing two pages of your site offtherecord.com. Feel to search "site:offtherecord.com" in google.

  1. offtherecord.com
  2. offtherecord.com/how-it-works

For the "how-it-works" page, Google is able to crawl this which requires a hash bang to get the content to render on the browser thereby requiring execution of JS and it works! However, it just doesn't seem to be able to crawl and/or index any of the other pages.

Putting https://offtherecord.com/how-it-works in the "Fetch as Google" tool causes a 301 redirect as expected to #!/how-it-works, however if I try to follow it in the tool then it truncates everything after #! url.

I've checked the Google crawler stats page on the webmaster tool and there are no crawler errors.

Similar threads:

  1. Google not crawling AJAX content: https://productforums.google.com/forum/#!topic/webmasters/_pdC55wUvfI;context-place=topicsearchin/webmasters/hash$20bang

  2. AmazonS3 + AJAX content: [stack exchange only allows 2 links for me]

We have html5 mode enabled in our AngularJs app via $locationProvider.html5Mode(true).hashPrefix('!');

Please advise on how we can address #1 and #2. We're looking into actually having a real web server if this is hurting our accessibility for search engine crawlers.

Thank you for your time

Improve this question
2
  • Hash bang crawling is now deprecated. Google still supports it for now, but they may not forever. Instead you should use pushstate to set real looking URLs for each page. Those are easier to deal with anyway because you don't have to create server site HTML snapshots.
    – Stephen Ostermiller ♦
    Oct 6, 2016 at 10:18
  • Thanks Stephen. pushState gets configured for this line $locationProvider.html5Mode(true) which I have above. Also, I got rid of the hashbangs URLs in the sitemap file in addition to 301 redirects on the S3 bucket. I cleared the CloudFront cache, and re-submitted the sitemap but only 1 URL was indexed which is the root domain. Good news is #1 is fixed and the google crawler can properly render the pages that werent being indexed. Bad news is that I submitted sitemap.txt and only 1 URL was indexed. I think the problem still relies on the fact that we don't have a real webserver
    – Mark M
    Oct 6, 2016 at 23:42

2 Answers 2

Reset to default
1

It seems you have a lot going on.

Your robots.txt needs to be looked at. I will leave it at that. Then...

  • Your text sitemap needs to be html for the site and xml for the search engine, not plain text.
  • Submit that sitemap to search engines (you can create your sitemap without the #! and it will work.)
  • Create a Webmaster Tools account for Google > add site > verify site submit Sitemap > Fetch as Google > Submit to Index.

You can also add the sitemap via htaccess

RewriteEngine On
RewriteRule ^sitemap\.xml$ /path_to_sitemap [L]

Make sure mod_rewrite is enabled

Improve this answer
4
  • Thanks Johnny. I assume you're hinting at missing the "User-agent" field in which I will add now. I assumed not having a User-agent would default to allow all user-agents so my mistake here. For the sitemap, google's crawler allows sitemaps to be in plain txt. See here: support.google.com/webmasters/answer/183668?hl=en We're using S3 as our "webserver" as explained above which is more like a file system which does not allow URL rewrites and only URL redirects.
    – Mark M
    Oct 6, 2016 at 1:40
  • @MarkM I realize they can be used but Google prefers you submit them as xml to their search and html on your site. If you can not get intended results, I was suggesting that you create the html or xml versions without the #! as the pages do exist without that in the url and then add the location in your htaccess so a bot is told of the sitemap at entrance and crawls it.
    – norcal johnny
    Oct 6, 2016 at 3:15
  • @Johnny, 1. I don't see where google prefers one format over the other as it states it supports both. However, I did try removing the hash bangs and submitted the sitemap to the webmaster tool but it only indexes the root domain. I'll remove the hash bangs anyways. 2. htaccess is an Apache-specific webserver feature but we're using AmazonS3 which is really not a webserver so this could be the problem. To state again, AmazonS3 does not support URL rewrites. Appreciate your input though!
    – Mark M
    Oct 6, 2016 at 20:00
  • So I got rid of the hashbang URLs in the sitemap and the site can function now without hash bangs. Good news is #1 is fixed and the google crawler can properly render the pages that weren't being indexed. Bad news is that I submitted sitemap.txt and only 1 URL was still being indexed. I think the problem still relies on the fact that we don't have a real web-server because S3 doesn't support URL rewrites
    – Mark M
    Oct 6, 2016 at 23:44
0

The answer to the two issues I was facing was addressed by moha297's answer here below. I didn't complete step #5 (setting up a web server) but pages are now being properly indexed.

https://stackoverflow.com/a/35354677/6929000

Thank you!

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.