Magento robots.txt rule also blocking JavaScript from WordPress

Question

I have a Magento site and a WordPress site is used in a directory for the blog. My blog keeps getting hit as not mobile friendly and the reason I believe is that there are certain resources blocked in robots.txt. The part of my robots.txt that is giving me problems is this:

# Do not index session ID     
Disallow: /*?SID=     
Disallow: /*?     
Disallow: /*.php$ 

This is from the standard Magento template. One of the blocked resource URL is: https://www.example.com/blog/wp-content/plugins/megamenu/js/maxmegamenu.js?ver=2.6

I think this line Disallow: /*? is blocking that example URL. I would prefer not to remove that line. Is there a way to allow everything in the wp-content directory while still blocking the Magento URLs with that rule?

Answer 1

What I understood is that you have a Magento website on https://www.example.com and a Wordpress website on https://www.example.com/blog You want to keep Disallow: /*? but at the same time allow the files with this pattern to be crawled in your Wordpress website under /blog

Your robots.txt should be like this

# Do not index session ID     
Disallow: /*?SID=     
Disallow: /*?     
Disallow: /*.php$
Allow: /blog/*?

Consider testing your robots.txt in the old version of Google Webmasters under Crawl -> robots.txt Tester