New answers tagged googlebot
There appear to be several criteria that can qualify a page as a soft 404: Redirect to the home page The phrase "not found" or equivalent in a prominent place on the page A blank or nearly blank page I don't know exactly how much text you need on a page for it to be considered non-blank, but that is certainly not the usual reason that pages are soft 404. ...
Mediapartners-Google is the user agent that Google uses to crawl pages with AdSense ads on them. The crawling is likely related to Ads that are shown by the video. Remove the ads and Google will stop trying to crawl like this.
You can use tags in your headers (<head></head>) to prevent crawling from most search engines: <meta name="robots" content="noindex"> They also specify that if you only want to block google specifically then you can use this: <meta name="googlebot" content="noindex"> Google throws up the same: ...
Bing absolutely does. Before I told it not to it would continually submit both a contact form and a survey form. I knew it was Bingbot because I included $_SERVER['HTTP_USER_AGENT'] in the $_POST information. No other spider ever submitted those forms.
They can and do. I've got a simple email collection form on the front of a site I'm about ready to launch. I just received a submission from that form from IP 22.214.171.124, which is a GoogleBot IP.
No, Googlebot doesn't use SPDY. SPDY was optimized for browsers, where things like pipelining & HTTP request headers are bottlenecks. These things generally aren't issues when crawling the web. That said, SPDY may have some benefit in saving the server some work (it can serve multiple URLs with one request/response), so I wouldn't say that it's ...
I find it plausible that the addition of cache headers would reduce the amount of crawling that Googlebot has to do. If you are telling Googlebot that the page can be cached for a week, then it may not come back to crawl again until that cache expires.
To prevent misuse the inner workings of Googlebot are kept secret so there is no real "expected behaviour". While searching the net I stumbled upon the following: ... search engines understand URL parameters but often ignore fragments. This implies URL fragments might get crawled. On another page I found a second hint. I always thought Googlebot ...
When you added CDN, did you also added site-wide SSL? If you did, your current site in WMT is HTTP and that won't be crawled anymore as you have move away from HTTP. So go ahead, add a new site in WMT as https:// in front of it and it will show you all the impressions and everything back to normal.
This error occur when your robots.txt file exists but is unreachable. Your site should return 200 HTTP status if the file exists or 404 if it doesn't, otherwise you would face such message from google. Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If ...
The base directory in your URLs will make no difference to Google. Google indexes pages linked from one directory to another. Google often chooses not to index URLs for other reasons: The page doesn't have enough link juice. If you have only one page that links to the page, it may not make the cut. The site and page is too new. Google often takes ...
You can add Robots Meta tags into your pages. It will tell robots to index the content of a page, and/or not scan it for links to follow. You can also put revisit meta tags, It will tell the spider to come back to your website and index it again. Here is the both tag
There are few ways to get indexed your post. Please follow mentioned below: Update XML sitemap in Webmaster Tools Fetch your post through Webmaster Tools Insert Meta index tag to your post Post new content regular basis
Google's crawl bots do not support TLSv1.2, this is fairly well known and people have ranted about Google's failure to support >TLSv1 for a while now as people have tried to make their website get a 100% score on places like SSLlabs -- which means they then get dropped off Google for unreachability. TLSv1.2 is substantially different from TLSv1. This is a ...
Top 50 recent answers are included