1

I moved a site from WordPress to Magento. But now Google webmaster does not index the site. I have submitted the sitemap, but it remains pending. I also have a robots.txt file which has the following content

User-agent: *
Allow: /

sitemap: http://www.example.com/sitemap.xml

Is there anything I have to do after moving from WordPress to Magento?

Edit :

I feel the problem is with robots.txt because in my crawl errors page in google search console, robots.txt fetch shows an orange status instead of a green tick. Under that tab, it shows a message 'Google couldn't crawl your site because we were unable to access your site's robots.txt file'. But when I go to mydomain.com/robots.txt, it displays my robots.txt file.

,I have tried Fetch as google in search console. It fetches completely when I use 'fetch' and partially when I use 'fetch and render'.

Improve this question
9
  • 1
    What about the URL structure? Has that changed? Are you 301 redirecting all the old URLs to the new? It would be preferable to use Disallow: (without any URL-path) instead of Allow: / (but that shouldn't make any difference to Google).
    – MrWhite
    Commented Mar 14, 2017 at 22:30
  • @w3dk , There is a slight change in URL structure in my website, but I don't have problem if the old url shows 'page not found'. (if that's what you were asking).I feel the problem is with robots.txt because in my crawl errors page in google search console, robots.txt fetch shows an orange status instead of a green tick. Under that tab, it shows a message 'Google couldn't crawl your site because we were unable to access your site's robots.txt file'. But when I go to mydomain.com/robots.txt, it displays my robots.txt file.
    – Abdulla
    Commented Mar 15, 2017 at 3:15
  • 1
    Google couldn't crawl your site because we were unable to access your site's robots.txt file. That just may be a temporary network error. Have you tried a Fetch as Google in Search Console? Check that to make sure G can see your home page at least.
    – closetnoc
    Commented Mar 15, 2017 at 3:40
  • @closetnoc ,I have tried Fetch as google in search console. It fetches completely when I use 'fetch' and partially when I use 'fetch and render'.
    – Abdulla
    Commented Mar 15, 2017 at 14:01
  • It is not uncommon that Google has network troubles accessing a site. This would of course effect being able to access the robots.txt. Generally, a few days later, all will work fine. I would be more concerned that rendering does not give you what you expect. You can also check your log files for errors. If you do a fetch, you can check your log file, however, it can take a while for the entry to appear because of caching. So keep that in mind. If G can access your site, then you are okay. Do not worry about it.
    – closetnoc
    Commented Mar 15, 2017 at 16:06

1 Answer 1

Reset to default
2

As W3dk says in the comments, a more correct robots.txt file would be:

User-agent: *
Disallow:

sitemap: http://www.example.com/sitemap.xml

But your biggest problem is most likely that you didn't redirect your old URLs to your new ones. You say:

There is a slight change in URL structure in my website, but I don't have problem if the old url shows 'page not found'.

Google is going to index pages right away if you redirect the old URLs to the new ones. Making this kind of change without redirects kills your inbound link profile, and Google's trust in your site. It may take months or years for your site to get indexed and enjoy rankings as good as it used to have. Implement the redirects.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.