0

When I google my domain, I can find it to be first but it's saying "No information is available for this page"

enter image description here

The website has meta information like this:

<meta name="description" content="My website description"/>
<meta name="keywords" content="keywords"/>
<title>Title</title>

<meta name="robots" content="INDEX,FOLLOW"/>

This is my robots.txt configuration:

User-agent: *

# Directories
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /setup/
Disallow: /update/
Disallow: /var/
Disallow: /vendor/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /wishlist/

# Files
Disallow: /composer.json
Disallow: /composer.lock
Disallow: /CONTRIBUTING.md
Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow: /COPYING.txt
Disallow: /Gruntfile.js
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /nginx.conf.sample
Disallow: /package.json
Disallow: /php.ini.sample
Disallow: /RELEASE_NOTES.txt

# Do not index pages that are sorted or filtered.
Disallow: /*?*product_list_mode=
Disallow: /*?*product_list_order=
Disallow: /*?*product_list_limit=
Disallow: /*?*product_list_dir=

# Do not index session ID
Disallow: /*?SID=
Disallow: /*?
Disallow: /*.php$

# CVS, SVN directory and dump files
Disallow: /*.CVS
Disallow: /*.Zip$
Disallow: /*.Svn$
Disallow: /*.Idea$
Disallow: /*.Sql$
Disallow: /*.Tgz$

Sitemap: https://www.mydomain.co.uk/sitemap.xml

I have a sitemap configured too. Looks like it's loading fine from the URL I included in robots.txt.

Can you see anything wrong?

  • Was the robots.txt file changed recently? What does Google's robots.txt testing tool say? – Stephen Ostermiller Apr 11 '18 at 16:36
  • I didn't change anything. It's a fairly new website but has been out for a couple of months now. The testing tool says "0 Errors | 0 Warnings" and when I request the homepage URL it's replaying "Allowed" – Greg Apr 12 '18 at 10:12
1

I'm not sure if your Head is a valid format:

<meta name="description" content="My website description"/>
<meta name="keywords" content="keywords"/>
<title>Title</title>

<meta name="robots" content="INDEX,FOLLOW"/>

I haven't seen it formatted in that way before.

I would change it to this:

<html>
<head>
<title>Title</title>
<meta name="description" content="My website description"/>
<meta name="keywords" content="keywords"/>

<meta name="robots" content="INDEX,FOLLOW"/>
</head>

I think the Title is supposed to be before the meta descriptions. Also make sure that this is all put into a HEAD tag in case it currently isn't.

For your robots.txt file, it all looks pretty good to me except I think it's possible that this could be confusing Googlebot:

Disallow: /*?
Disallow: /*.php$

If I'm not mistaken /*? should only disallow URLs with ? parameters in them. But perhaps it's preventing Google from crawling your entire site.

/*.php could maybe be confusing Google if your homepage is index.php

I would remove those 2 lines of code unless they're completely necessary. Or change them to something more specific if possible.

After they've been removed and the homepage has been recrawled, see if it corrects it.

  • 1
    Hi Michael! Thanks for the answer. Head seems to have a valid format. I have compared it against different magento 2 stores and they all seems to have this formatting. Of course the meta info and title are located between <head> tags. I have commented out the two entries you have mentioned. Will see if that solves the problem. Will let you know how it went. Up-vote from me :) – Greg Apr 12 '18 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.