New answers tagged googlebot
2
To further investigate this problem you should:
Look at your server log files. Find the requests from Googlebot and verify that they are resulting in 301 status. If you direct Googlebot to hit your server and there is no request in the server log, then you have a DNS problem. Your domain name is not pointing to the server that you think it is pointing ...
2
Google sends out a notice whenever it has issues connecting to your site.
The message will be similar to:
Over the last 24 hours, Googlebot encountered 205 errors while
attempting to connect to your site.
I've seen it send notices for outages less than 1 hour but still reference the 24 hour window. If you only received one message on one day, then ...
0
A CDN service could sit in front of your site and filter out known crawlers; they also filter out spammers and since they cache your images at locations around the world your site would be faster.
I have been using CloudFlare for about a month on a site for a client and they have seen a decrease in bandwidth use, and an increase in traffic. CloudFlare also ...
4
The Googlebot can submit forms, but it generally doesn't unless it can detect a reason to do so.
So from the links, if your translations were AJAX'd and built properly, Googlebot may very well submit the form to see what the results would be. However, this behavior (especially on POST), is not guaranteed and you should probably use GET to make things more ...
0
The simplest solution would be to disable the 'direct' subdomain. If however you want to use that subdomain you would have to use a more creative approach.
One way to do it is to have a dynamic robots.txt. When the web spider requests robots.txt we redirect it to our dynamic robots page. If the subdomain matches our criteria we send a 'disallow' otherwise ...
0
To cover your last point first, if you have 97k URLs indexed but 1.2m URLs found with a parameter, then Googlebot is ignoring many URLs (over 1.1m in fact).
As for the main question, Webmaster Tools states when you set the option "No URL" that they may remove URLs from the index - so there is no guarantee they will be removed.
You should go through those ...
0
Your robots.txt tells all bots (User-agent: *) not (Disallow:) to crawl any URL starting with a slash (/), which would be all URLs, of course.
If you want to allow all bots crawling all your pages, remove the slash:
User-agent: *
Disallow:
Note that Allow: is not part of the original robots.txt specification, however, some/many crawlers recognize it. ...
1
Your robots.txt own these lines:
User-agent: *
Disallow: /
It indicates to Googlebot and other search spiders to not index your website.
Just put these lines instead and wait:
User-agent: *
Allow: /
That way, all search spiders (Googlebot included) will index your website.
0
I can confirm that your robots.txt and HTML are not blocking Google from scanning your site, so whats the problem? In short... time.
Google may state that it takes two days for Google to take note of a robots.txt but in no way does this imply that all these pages blocked previously will be unblocked and instantly crawled.
Now that Google can crawl your ...
-1
If you have the list of those URLs then you can delete those URLs with of help of Google Webmaster tool. Visit the link for complete pictorial description: http://infoheap.com/how-to-remove-urls-from-google-index-using-webmaster-tools/
0
I do not recommend keeping same content in all sub-domain as it is against search engine's guidelines and may result in poor performance.
If you are operating different country, you can have same content but still you will have to change the offer price/services location etc. To redirecting crawlers is not a good practice as there is not country specific ...
0
To get Googlebot to crawl faster you need:
A fast server. The less time that it takes for Googlebot to download each page the faster it will crawl. Don't worry about the images, css, and javascript. Just serve the html faster. I find that enabling gzip compression helps. Cache and preload some of your site's data if appropriate. Shrink your page ...
0
I'm seeing the same issue across multiple dedicated servers in different datacenters.
Matt Cutts posted on his blog that this looks like a Google issue. I wouldn't freak out unless it continues for another couple days.
0
Is a daily crawl from the Googlebot a good thing or bad thing?
Good/Bad will depend on ...
Does the content change daily ?
Can you afford the bandwidth/transfer bill ?
Is it the whole site or specific pages ?
Have you solved any canonicl and duplicate issues ?
And of course, far more importantly ... is it translating into more visitors and more ...
Top 50 recent answers are included
