Hot answers tagged googlebot
13
I would strongly recommend registering your site with webmaster tools. There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google.
...
6
The Google help page on submitting sitemaps says:
"Note: If you are providing a Sitemap index file, you only need to issue a single HTTP request that includes the location of the Sitemap index file; you don't need to issue individual requests for each Sitemap listed in the index."
Assuming that you're indeed providing correct lastmod timestamps in your ...
6
Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them.
This will depend a bit on the exact implementation of the crawlers robot.txt ...
5
The most probable reason is that the pages won't return a 403 header.
You can check that using the Web Developer Toolbar in Firefox or Chrome. The tool is located under "Information" -> "View Response Headers".
Also, the way I create my error pages is:
I create some dummy error page. Let's say 403.php.
I create an actual error page. For example ...
5
Use google sitemaps. We use them where I work, you can specify how often pages change, etc:
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184&from=40318&rd=1
You cannot guarantee that they will crawl today, but for a small enough site, it will help them find your content faster.
5
Usually no but you have to be really sure that the URL is not present anywhere on the web, in your sitemap if you are publishing it. You also have to be careful on web server access statistics if you make them public.
In addition You can always use the robot.txt file to tell Google not to crawl the URLs.
But this is just security by obfuscation, if you ...
5
Yes, Google will find it somehow!
They monitor people's browser/search history via Google Accounts/Toolbars/Social Networks and the like - then use that data to augment and prioritize their crawler.
So if a user visits your page while logged into a google account with it's search history tracking enabled Google may find out about your page. You also ...
4
It's a bad idea to hide elements for google, google is smart and you could definitely get punished for this.
Read more here: http://www.google.com/support/webmasters/bin/answer.py?answer=66355
4
First, stop submitting your sitemap every day. You can keep updating it of course, and Google will periodically check it for updates. But as long as your site is crawlable, that's more important. You can also submit your RSS feed which will help Google find your newer content.
Regarding the "catalog", I think how you are currently doing it is fine, as long ...
4
I just realized that the user must have found an outbound link on this authenticated page, and then leaked the private URL as Referer when clicking through to some other website. This is the only possible explanation, and should really have been obvious from the start.
Once leaked, the private URL may have been exposed to Google in a number of ways, e.g. ...
4
Have you tried to view your site like a bot would? Xenu's Link Sleuth is a great way to look for bad links and should report links like you have above in its report. That should tell you where your problem lies.
4
First, the "Allow" directive in your robots.txt does nothing as robots spider everything by default.
Blocking robots from *.* is probably OK in some situations, but remember that you are blocking every URL that simply contains a dot. A more reliable method may be blocking individual extensions (if there are not too many) eg *.html and *.php on separate ...
4
It's not the technique, it the reason why you're doing it that matters. Based on your question you're clearly doing this to manipulate your search rankings. So whether you use JavaScript or CSS it's still cloaking and still against the terms of service of the search engines. Cloaking gets you banned if you get caught.
So, instead of trying to cheat the ...
4
It looks like you have a significant performance problem. Somehow I doubt that you want bots to not index your content, so I'm just not going to go there...
Absolutely the first thing you should do is set up caching. W3 Total Cache is a good start. I used to use WP Super Cache, but the former has many more options for setting up caching in various different ...
4
That is what the robots meta tag is for, control per page for indexing and following.
I've come to prefer it over using robots.txt as it gives finer control.
For your page, you'd want noindex,follow for the setting. The robot will read the page, not index it, but follow all the links off the page.
<meta name="robots" content="noindex,follow" />
4
Your robots.txt file does not appear to conform to the robots.txt specification (you must specify the user agent before the directives and I do not believe that Noindex is a valid directive).
Consider the following updates:
User-agent: googlebot
Disallow: /Office/LocationDetails.aspx
Disallow: /office/default.aspx
Disallow: /Electronics/Communitylist.aspx
...
4
The Googlebot can submit forms, but it generally doesn't unless it can detect a reason to do so.
So from the links, if your translations were AJAX'd and built properly, Googlebot may very well submit the form to see what the results would be. However, this behavior (especially on POST), is not guaranteed and you should probably use GET to make things more ...
3
According to Google:
In general, we don't follow them. This means that Google does not
transfer PageRank or anchor text across these links. Essentially,
using nofollow causes us to drop the target links from our overall
graph of the web. However, the target pages may still appear in our
index if other sites link to them without using nofollow, or ...
3
If domain name is new (you recently bought it) then Google may still try to get old URLs (not because it remembers those links (it will de-index such not-working-for-a-long-time links) but because there is still a site that has links to this domain)
Wherever those links were published there maybe a typo in URL (domain name) and they are pointing to your ...
3
Yes. The proximity of text near a hyperlink is almost certainly a factor in search engine rankings.
Your example is still good to do even without keyword proximity because it helps to ensure they rank well for their company name. This is never a given and very important when your name is similar to another company or web pages with bad press about your ...
3
It's a good thing. This means that Google will find your new content faster which means it can be included in their index sooner. It's got to be in the index to be found in searches.
Keep in mind that this doesn't mean your pages will rank well. It doesn't even necessarily mean your pages will be indexed. But under normal circumstances being visited ...
3
Why do your URLs churn so much? You write:
"[I]n webmaster tools I see a lot pages that were submitted months ago, currently deleted, and still in index even thought I retrieve 404 with noindex option."
If you keep your pages up only for a month or two before deleting them, it's no wonder that Google can't keep up.
In particular, keep in mind that ...
3
From the time you see Googlebot hit your site in your server access logs, to the time it's updated the index, and the time it has updated their cache the dates will vary. It depends where they are in the cycle of pushing out an update to index and cache servers. Sometimes it can be within minutes other times it can take a ~week
This is something you just ...
3
Absolutely!
Any time you have pages that are substantially similar or even identical (through URL issues like www v no-www, https v http, query-string permutations etc), you are forcing the search engine to choose which version it should display in the search result.
If different versions of those pages have received different back-links, they are not ...
3
If the majority of your content is behind a login then bots won't have access to it which will limit the amount of time they spend on your site.
Good bots, like those from major search engines, will automatically set their crawl rate to one appropriate for the site. Although there are some stories of bots hammering sites they tend to be older and you don't ...
3
Google has written a fair amount on the recommended way to present multilingual content: http://googlewebmastercentral.blogspot.com.au/2011/12/new-markup-for-multilingual-content.html
They also have a fair amount of detail in terms of implementation on this subject:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=182192
3
How to Keep your Rankings
You mentioned that your going to keep the same URLS, the same metas and the same title tags while this is a good start the content is the KEY element which can effect your rankings. The Title Tags, Metas and URLS are only siginals that Google uses to estiblish more about your page so effectly changing the content you will change ...
2
Yes, Google will obviously cache robots.txt to an extent - it won't download it every time it wants to look at a page. How long it caches it for, I don't know. However, if you have a long Expires header set, Googlebot may leave it much longer to check the file.
Another problem could be a misconfigured file. In the webmaster tools that danivovich suggests, ...
2
If you want fast and guaranteed crawling, create a google adsense account and ad the google ad to the source code of your site. You can do this just until they spider the site, and you can probably do it on pages that simply link to your important content, and which only spiders are otherwise going to be likely to find. Since they advertise based on ...
Only top voted, non community-wiki answers of a minimum length are eligible
