I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good). Is it good or bad, something like ...
I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...