3

I am trying to index a page which was set to "discourage search engines to index" until 2021-10-31. Now it's 2021-11-06 and Google does not even try to access the page.

If I use the search console, it get errors it cannot be fetched. Same with the mobile testing tool:

URL is not available to google

My page meta is content:

<meta name='robots' content='max-image-preview:large' />

https://page.et/robots.txt

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
  • There was a forward from page.et/robots.txt to page.et/en/robots.txt which I also disabled. In my Chrome it's still cached, but using an incognito tab I can see the robots.txt

  • When trying to request the page in Google I see zero evidence of Google trying to access my page, not even the robots.txt - in my web server logs.

Is Google really blocking all requests to my page and storing this persistently for more then 4 days ? Or is there maybe really a technical reasons it does not access my page?

Are there any additional tools to debug this?

Update: In the search console, it displays this:

Index request rejected

URL is not on Google

How do I read these reasons?

Edit 2021-11-07 Same result .. I don't think it's just a caching problem by Google :-( I mostly wonder how I can debug this properly. The output by Google search console does not seem helpful :-(

Edit: I am getting some strange errors in Google Pagespeed which I tried to see if Google can access the page at all. Following up eventual server configuration errors here: https://serverfault.com/questions/1082838/pagespeed-cannot-always-access-page-failed-document-request-ssl-certification

12
  • 3
    Replace your robots meta with this and try again. To provide a good answer we need some things ruled out. <meta name='robots' content='index, follow, max-image-preview:large' />. Make sure it's in the <head> of course. Nov 5 at 17:59
  • 3
    Google should index by default but in your case it's not even crawling and you're not blocking it in your robots.txt. You said in the past the site was set to no-index right? This is a quick thing to rule out buggy behavior I have seen in the past. Please make sure all caches are disabled for now. This is not an answer. As of now there are too many unknowns to provide one. Nov 5 at 20:34
  • 1
    What is your stack? Would like to know everything. Ex: Ubuntu, Nginx, Cloudflare Nov 22 at 5:20
  • 1
  • 1
    On the server what is the output of the following command: stat -c "%a %n" /var/www/your-site/robots.txt? Replace /your-site/ with your sites directory name of course. Nov 23 at 15:57
3
+100

.Et domain zone NS block Googlebot

The problem is on the side of the .et name servers - they "block" Googlebot.

This Google Support thread reports a problem with indexation with all domains in ".et" zone

13
  • Do you have a source for your quote in English? It would be more useful to others.
    – Trebor
    Nov 23 at 23:01
  • 1
    @Trebor translate the page? This is the answer. Nov 24 at 1:43
  • 2
    Page translation: "At the moment, I only know that the problem is on the side of the .et nameservers - they "block" Googlebot ... Google specialists are trying to contact representatives of NS .et to resolve the current situation." Nov 24 at 2:07
  • @MikeCiffone it's completely illegible characters for me. Can't even find a translate button.
    – Trebor
    Nov 24 at 3:25
  • 1
    @Alex, A DNS block works by removing the IP address name from the “phonebook” on your server. For example, if you were to block Twitter using a DNS blocker, your server would intentionally forget the name assigned to 199.59.149.165. As a result, blockers disable your server from locating particular web pages. To block whole genres of web pages, like piracy sites, DNS blocking services can set your server to forget large swaths of IP addresses that fit certain criteria. List of Googlebot IP addresses developers.google.com/search/apis/ipranges/googlebot.json
    – skill
    Nov 25 at 17:11
1

okay. Let's be technical. It's really good that you've checked the access log for google accessing your robots. That's important. Our first aim would be to force google to index robots. But before we do it, make sure you actually Allow that page in your robots. Just to override any conflicts there.

Now see this? https://developers.google.com/search/docs/advanced/robots/submit-updated-robots-txt I know, I know, it's trivial, but no, scroll down where it says If you need to update the cache faster, use the Submit function of the robots.txt Tester. Use it!

Okay, now our real problem may be not in robots.txt. So that was attempt 1. Let's go to attempt 2.

Do the dofollow meta set up the first comment from Mike suggests. Now take the page G doesn't wanna crawl and add some random query param to it. Like page.et/blah-blah/?foo=bar&someparam=something and try submitting it. Your backend will ignore a random query param and serve the same page. From G's perspective, however, it would be a different page. Now I hope G eats it. We're debugging it now.

Now if G eats it, we're one step closer to resolving the issue. Well, even on it's own, it can resolve it, but still, try using... uh... try making sure that on this ?someparam version of the page the canonical is set to the original page with no ?someparam. That would be an interesting way to solve it.

Also, keep monitoring the hits to all three urls from google. They give a lot of insight into G's behavior.

You can... force G to your page by redirecting it from your ?someparam page, hehe. That would be funny to watch, but don't force it yet. It's interesting to see if more civilized ways work.

2
  • I tried adding random query strings, the page can still not be fetched in the search console.
    – Alex
    Nov 23 at 6:39
  • Which means that it's not cache. It's something different. Have you added the allow to robots.txt prior to testing?
    – BNazaruk
    Nov 23 at 17:04
1

When testing with Google Lighthouse, I receive a network error: `

"DevTools failed to load source map: Could not load content for https://page.et/wp-content/plugins/map-block-leaflet/lib/leaflet.markercluster.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE"`

Additionally, under the Lighthouse Issues tab there's the message:

A page or script is accessing at least one of navigator.userAgent, navigator.appVersion, and navigator.platform. Starting in Chrome 101, the amount of information available in the User Agent string will be reduced. To fix this issue, replace the usage of navigator.userAgent, navigator.appVersion, and navigator.platform with feature detection, progressive enhancement, or migrate to navigator.userAgentData. Note that for performance reasons, only the first access to one of the properties is shown. 1 source leaflet.js:1

I'm thinking you have a Wordpress plugin or piece of code relating to your map that is causing your problem. It's either being actively blocked or the file is missing. I tried accessing the file above and received a 404 error from Firefox.

I would disable any plugins and retest. Also, do you have a firewall or security plugin that could be blocking Google?

3
  • I believe a .map (sourcemap) file is only necessary for debugging into javascript files, or am I wrong?
    – Alex
    Nov 23 at 6:35
  • I don't believe that is the reason, especially because Google not even tries to load the page, according to the access log - so it should not be caused by a Javascript problem (especially there is not a real one)
    – Alex
    Nov 23 at 6:36
  • @Alex, you may be correct about your map file, but it appears that at least one of your plugins is calling it. However, the 404 error could be an indicator of another problem, like something blocking access.
    – Trebor
    Nov 23 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.