Tag Info

New answers tagged

2

As I mentioned in my comment above, if the robots meta tag has been on these pages all the time AND the pages are not blocked by robots.txt (which prevents crawling, but not indexing) then these pages should not get indexed. Digging a little deeper, these pages do appear to be fully "indexed", with a complete description in the SERPs. So they have not been ...


1

I've seen this before and I think it's caused by a vicious circle! If you are blocking pages from being crawled by Google in robots.txt then Google cannot access the page to see the NOINDEX tag, so the pages will not be removed from the index if they had already got indexed Blocking pages in robots.txt will stop Google crawling them, but it won't stop them ...


2

I think the issue could be locally at your end because I see a robots.txt file at the location that looks like the following:- sitemap: http://www.eonclinics.com/sitemap.xml User-agent: * # disallow all files in these directories Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /archives/ disallow: /*?* ...


1

I'd go with the canonical header. The purpose of it would be to tell search engines the preferred URL where the content should be indexed (so you can reap the benefit of any traffic those images send your way). Where the content should be indexed is not the same (in this case) as where it is served from and a canonical header seems like the appropriate way ...


0

Usually, this line disallow inner search results from crawling. The best way to prevent site from crawling is to close it with the password (custom authorization).


2

Disallow: *?s= Bots following the original robots.txt specification would not be allowed to crawl URLs like these: http://example.com/*?s= http://example.com/*?s=foo http://example.com/*?s=/ So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path). But many bots use (their own) extensions to the ...


1

If you want to disallow all robots to crawl your site simply use: User-agent: * Disallow: / User-agent: * means that all robots should follow the rule that comes next. And Disallow: / prevents them to crawl any path. You can see more here on robotstxt.org. I think your Disallow: *?s= means that robots are not allowed to crawl URIs with parameters, but I'm ...


-1

The robots.txt doesn't define mobile friendliness, it is supposed to define what robots are not allowed to access certain resources. In your case, you're preventing every robot from accessing a set of folders except for one CSS file. When google checks a page, it tries to access all assets in that page. Go on that page in a browser and choose to view the ...


1

I would pare that down to: Disallow: /search/ Sitemap: http://example.com/sitemap_index.xml You should certainly remove wp-content from that list. That has your themes, plugins, and images. For mobile friendly you want all these crawled. If you want more info on what each of the other pieces do, here is a guide to the WordPress directory structure.



Top 50 recent answers are included