2

The SDTT will not read my pages. No matter which subpage I try to fetch, I get the error:

The URL is unreachable. Ensure robots.txt is accessible and the server is responding with a 200 status code.

An example page is: https://www.rsvp.dk/events/del-popolo-secret-adventures

All pages are loading fine (200 status). Other sections of my site will test fine, but all pages under /events (the only ones I really care about!) will not load. I didn't have a robots.txt file, but I tried adding one, and testing it for those URLs shows they're allowed.

The page at /events used to be an Angular app that loaded list data and details pages asynchronously, but it was changed about a month ago to plain old HTML pages. The new pages are being indexed somewhat, but not as well as I'd want them to, and structured data (JSON-LD) is not being picked up. I don't know if the previous Angular page could still somehow be affecting Google?

For what it's worth, if I copy/paste the source into the window, it verifies fine, so it seems that Google for some reason just can't access them?

UPDATE: Since some sections of the site, such as /blog can be accessed fine, it can't be an SSL or blocking issue. The strange thing is that I can create a new URL for the same content, replacing /events with /popups. This new path still can't be reached. I'm really lost here...

  • I can confirm that your ssl, robots and page statuses are correct. What the community can't confirm is if your hosting is blocking the fetch. I'm guessing that your hosting has blocked the IP that Google uses, look through your access/errors logs, if its empty then contact your hosting as they might be blocking before the request even reaches your server. – Simon Hayter Aug 2 '15 at 11:06
  • Thanks @simon, but SDTT is able to reach other areas of my site, such as https://www.rsvp.dk/blog only the /events path is unavailable. There's no physical folder, and no access restrictions on my side. The site is hosted on Azure, so there's no blocking on that side either. – mrlund Aug 2 '15 at 13:59
2

Case solved. For anyone else with a similar issue later, I had an i18n culture resolving mechanism that is based on cookies. If no preference cookie is set, it'll do a geolocation lookup based on the host IP address, and set a cookie according to the country of origin of the request.

I'm not sure if it's the simple act of setting a cookie on the response that caused the issue for SDTT, or some kind of problem resolving the IP address and associated country, but not setting a cookie in the response solved it, so that might be the issue.

I had a fallback in place, so getting a strange response from geolocation lookup should have been handled, also, I've tried running it locally by spoofing a number of IP addresses of the googlebot (according to http://chceme.info/ips/) but these all behaved as expected and returned 200. I can't be certain there isn't a bug in it somewhere, but right now my money's on the response including a cookie.

1

I tested your site with the command line tool CURL version 7.16.2 with no useragent string and the googlebot user agent string and it produced the following results:

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). The default
bundle is named curl-ca-bundle.crt; you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).

What I would suggest then is for you to make sure you are using valid SSL certificates and that your webpages load fast from data centers near Google. You can do this at the webpagetest.org website.

If that fails, then I'd recommend not using SSL for all your generic informative pages and use it only for pages that involve sensitive data such as when users are logged in or are filling out a form containing sensitive data.

  • thanks @mark , i'll chase that up with the SSL certificate issuer, but it doesn't seem that's the problem. I created an alternative URL at: http://www.rsvp.dk/popups/del-popolo-secret-adventures and removed the SSL requirement, but even without https, and with the new URL (/popups/ instead of (/events/), the problem persists. – mrlund Aug 3 '15 at 10:05
  • SSL shouldn't be the issue. I got this response from the issuer: 'You can check the installation using SSLchecker.com tool: sslchecker.com/sslchecker?su=05d6400960ee488f8a4016fcb1ac9df7 It shows that CA root certificate is missing due to the peculiarity of Microsoft IIS servers, which do not send the root certificate during the SSL handshake. Usually, this does not cause any issues as most of the modern browsers do have the root certificates of all major Certificate Authorities in their internal storages.' IIS not sending root certificates shouldn't cause an issue with SDTT? – mrlund Aug 3 '15 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.