New answers tagged

1

Allow: * should actually be be Allow: /. That could be the issue... because, really, using Allow: is somewhat meaningless by itself -- its purpose is for allowing a sub-path within a denied path. As it stands, it seems possible that your file is being misinterpreted. Anything not denied is supposed to be implicitly allowed. If you want to allow ...


0

XML is the markup language (Extensible Markup Language) and gz (short for GZip) is the compression format in this case. All sitemaps must be in xml format like sitemap.xml or in a compressed xml format like sitemap.xml.gz. Also your sitemap's url should be the very first line in your robots.txt file and it should read: Sitemap: ...


1

Disallow: /folder/file.aspx What you had in the beginning is all that's required - in order to block both /folder/file.apsx and /folder/file.aspx?foo=bar. If in doubt check it with the robots.txt testing tool in Google Search Console. robots.txt is prefix matching, so there is never any need to include the wildcard (*) at the end of the URL path. The ...


0

As long as some of the query strings don't need to be crawled you can simply disallow access to the entire file using Disallow: /folder/file.aspx* This will disallow crawling both on the file on its own as well as all query strings appended to the end of the file.


3

I have them disallowed in my robots.txt... but it doesn't change anything, Google still show them, also on those pages contact, services,... I use Noindex from Yoast. Still no result. So what can I do else? Or maybe I do something wrong If you specifically don't want these pages indexed then you should use the noindex robots meta tag in the head section ...


1

The Google AdsBot does not only check one level deep to assess the page and it is not simply for assessing if the page is a good landing page or not. The AdBot has multiple roles, firstly to check the actual landing page for the ad to see what is on the landing page, this improves the quality of ad delivery to end users by more accurately targeting ads to ...


1

We have also faced similar issue on our website. This happened because initially you must have disallowed your home page in robots.txt. Now even if you allow, it takes sometime around a week-two to reflect in google indexes. I have tested your robots.txt and it looks fine.


0

There is no need to block the page in its entirety, what you can do is apply a change to your site so that if no parameters are passed then the site returns a 404 not found error (important this has to be a http code and not just printed on the screen) and allow Google to spider your results page. Why do this? Well blocking any spider access to the results ...


1

My hypothesis on the 5429 errors is that Google is trying to parse an HTML doc, (i.e., the redirect is working, but ending up in the wrong place). Could be a 404 page, an error page or even the home page. I had this very problem yesterday where the http://example.com/robots.txt was redirecting to /index.php and then again to my home page due to a dodgy ...



Top 50 recent answers are included