1

I have a Magento site and a WordPress site is used in a directory for the blog. My blog keeps getting hit as not mobile friendly and the reason I believe is that there are certain resources blocked in robots.txt. The part of my robots.txt that is giving me problems is this:

# Do not index session ID     
Disallow: /*?SID=     
Disallow: /*?     
Disallow: /*.php$ 

This is from the standard Magento template. One of the blocked resource URL is: https://www.example.com/blog/wp-content/plugins/megamenu/js/maxmegamenu.js?ver=2.6

I think this line Disallow: /*? is blocking that example URL. I would prefer not to remove that line. Is there a way to allow everything in the wp-content directory while still blocking the Magento URLs with that rule?

2

What I understood is that you have a Magento website on https://www.example.com and a Wordpress website on https://www.example.com/blog You want to keep Disallow: /*? but at the same time allow the files with this pattern to be crawled in your Wordpress website under /blog

Your robots.txt should be like this

# Do not index session ID     
Disallow: /*?SID=     
Disallow: /*?     
Disallow: /*.php$
Allow: /blog/*?

Consider testing your robots.txt in the old version of Google Webmasters under Crawl -> robots.txt Tester

  • Thank you I will give that a try. Does it matter if the allow is before or after the disallow? I am trying to figure out precedence. – tjjen Apr 30 at 21:23
  • In terms of Google, the order of precedence does not matter – Shahzeb Qureshi May 1 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.