Hot answers tagged

13

I would strongly recommend registering your site with Google Search Console (previously Google Webmaster Tools). There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


13

Google does index XML sitemaps (like any XML file). If Google is aware of a URL and it returns a valid response then it's going to pass Google's inclusion rules and could get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. The recommended method to ...


12

Several other search engines (Bing, Yandex, etc.) still use the _escaped_fragment_ system. They're not going to stop using it overnight just because Google has. Thus, if you care about your site being indexable by search engines other than Google, you may want to still support this scheme. Certainly, if you already have set up support for ...


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


10

A similar question was asked on Stack Overflow back in January. John Mueller was kind enough to respond with the following: "googleon" and "googleoff" are only supported by the Google Search Appliance (when you host your own search results, usually for your own internal website). They are not supported by Google's web-search at all. As far as ...


10

That's a very bad idea and your site will suffer in the organic search rankings. For one, Google does have image recognition abilities and so your assumption that the bot can't "view" the image is wrong. And two, the algorithm does render pages to decide whether they are not mobile friendly and if your pages are delivering different content to users and to ...


8

Yes, if the site requires authentication then robots will not be able to crawl it. You can also specify a robots.txt file so that the entire site is disallowed


8

It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming. Check that the I.Ps making these request are owned by Google.


8

Great question, and one many webmasters might be interested in since the Baidu spider is notoriously aggressive and can zap resources from servers... As indicated in Baidu's Web Search news, the Baidu spider does not support the Crawl-delay notification setting, and instead requires you to register and verify your site with its Baidu Webmaster Tools ...


8

No, Googlebot doesn't use SPDY. SPDY was optimized for browsers, where things like pipelining & HTTP request headers are bottlenecks. These things generally aren't issues when crawling the web. That said, SPDY may have some benefit in saving the server some work (it can serve multiple URLs with one request/response), so I wouldn't say that it's ...


7

The Googlebot can submit forms, but it generally doesn't unless it can detect a reason to do so. So from the links, if your translations were AJAX'd and built properly, Googlebot may very well submit the form to see what the results would be. However, this behavior (especially on POST), is not guaranteed and you should probably use GET to make things more ...


7

The best option is to have a noscript fallback to regular pagination. The search engine will pick up on the links in the noscript section and index those pages. This has the added benefit that anyone browsing your site without JavaScript enabled will get a better user experience.


7

You have nothing to worry about. You can use display: none; to switch menus. Search engines are much better at understanding JS and CSS. As long as you are not intentionally trying to manipulate things to get a better ranking. Using display: none; to hide big blocks of text will get you penalized. So if you are only using to hide your desktop menu on ...


6

It's hard to say for sure but here are likely scenarios: The user has a browser toolbar or extension installed that reports the URLs they visit to Google. Someone linked to that URL and Google found it by crawling the page with that link on it.


6

The Google help page on submitting sitemaps says: "Note: If you are providing a Sitemap index file, you only need to issue a single HTTP request that includes the location of the Sitemap index file; you don't need to issue individual requests for each Sitemap listed in the index." Assuming that you're indeed providing correct lastmod timestamps in your ...


6

It's not the technique, it the reason why you're doing it that matters. Based on your question you're clearly doing this to manipulate your search rankings. So whether you use JavaScript or CSS it's still cloaking and still against the terms of service of the search engines. Cloaking gets you banned if you get caught. So, instead of trying to cheat the ...


6

Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them. This will depend a bit on the exact implementation of the crawlers robot.txt ...


6

w3d's answer about using X-Robots-Tag appears to be the correct way to do this. Here is code that can be used in .htaccess or Apache configuration files to do so: <Files ~ "sitemap.*\.xml(\.gz)?$"> Header append X-Robots-Tag "noindex" </Files> Reference: WebmasterWorld - Sitemaps showing up in SERP - How to prevent this?


6

Google does not index data URI images for Google image search. Google's John Mueller says so here and in the comments below. Because data URI images are not indexed in Google image search, the EXIF data in them is irrelevant. You can verify that these images are not indexed. I searched Google images for "data uri" and spot checked the results. All of ...


6

robots.txt can block JavaScript files from Googlebot. http://www.robotstxt.org/ has more information about how to construct a robots.txt file. You could put your JavaScript that shows the password into an external JavaScript file (called showlists.js): $(document).ready(function(){ showLists(); }); Call that JavaScript file in the page head: ...


6

Here is what Google's John Mueller says about Noindex: in robots.txt: We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all. deepcrawl.com has done some testing of the feature and discovered that: It still works ...


5

Use google sitemaps. We use them where I work, you can specify how often pages change, etc: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184&from=40318&rd=1 You cannot guarantee that they will crawl today, but for a small enough site, it will help them find your content faster.


5

The most probable reason is that the pages won't return a 403 header. You can check that using the Web Developer Toolbar in Firefox or Chrome. The tool is located under "Information" -> "View Response Headers". Also, the way I create my error pages is: I create some dummy error page. Let's say 403.php. I create an actual error page. For example ...


5

You can do this with section targeting: <!-- google_ad_section_start(weight=ignore) --> [stuff to ignore here] <!-- google_ad_section_end -->


5

Usually no but you have to be really sure that the URL is not present anywhere on the web, in your sitemap if you are publishing it. You also have to be careful on web server access statistics if you make them public. In addition You can always use the robot.txt file to tell Google not to crawl the URLs. But this is just security by obfuscation, if you ...


5

Yes, Google will find it somehow! They monitor people's browser/search history via Google Accounts/Toolbars/Social Networks and the like - then use that data to augment and prioritize their crawler. So if a user visits your page while logged into a google account with it's search history tracking enabled Google may find out about your page. You also ...


5

Google has written a fair amount on the recommended way to present multilingual content: http://googlewebmastercentral.blogspot.com.au/2011/12/new-markup-for-multilingual-content.html They also have a fair amount of detail in terms of implementation on this subject: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=182192


5

Unfortunately no one can say you when Google will. According to my experience it can take a lot of time, also several months.


5

Use the Google Webmaster Tools and turn down the Google Crawl Frequency. Sign in to Webmaster Tools > Configuration > Crawl Rate



Only top voted, non community-wiki answers of a minimum length are eligible