Tag Info

Hot answers tagged

13

I would strongly recommend registering your site with Webmaster Tools. There is a crawler access section under site configuration that will tell you when your robots.txt was last downloaded. The tool also provides a lot of detail as to how the crawlers are seeing your site, what is blocked or not working, and where you are appearing in queries on Google. ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


11

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless ...


8

It's very simple to spoof a User-Agent header and pretend to be a Google Bot. But much harder to fake the I.P. from which the request is coming. Check that the I.Ps making these request are owned by Google.


8

A similar question was asked on Stack Overflow back in January. John Mueller was kind enough to respond with the following: "googleon" and "googleoff" are only supported by the Google Search Appliance (when you host your own search results, usually for your own internal website). They are not supported by Google's web-search at all. As far as ...


7

Yes, if the site requires authentication then robots will not be able to crawl it. You can also specify a robots.txt file so that the entire site is disallowed


7

The best option is to have a noscript fallback to regular pagination. The search engine will pick up on the links in the noscript section and index those pages. This has the added benefit that anyone browsing your site without JavaScript enabled will get a better user experience.


6

It's hard to say for sure but here are likely scenarios: The user has a browser toolbar or extension installed that reports the URLs they visit to Google. Someone linked to that URL and Google found it by crawling the page with that link on it.


6

The Google help page on submitting sitemaps says: "Note: If you are providing a Sitemap index file, you only need to issue a single HTTP request that includes the location of the Sitemap index file; you don't need to issue individual requests for each Sitemap listed in the index." Assuming that you're indeed providing correct lastmod timestamps in your ...


6

Robots that do not recognize wildcards (which is not in the official spec) will treat * as a literal character. The fact that it is not a valid URL character may mean that they ignore the rule altogether. In either case, it likely means that the rule will have no effect on them. This will depend a bit on the exact implementation of the crawlers robot.txt ...


6

Great question, and one many webmasters might be interested in since the Baidu spider is notoriously aggressive and can zap resources from servers... As indicated in Baidu's Web Search news, the Baidu spider does not support the Crawl-delay notification setting, and instead requires you to register and verify your site with its Baidu Webmaster Tools ...


5

The most probable reason is that the pages won't return a 403 header. You can check that using the Web Developer Toolbar in Firefox or Chrome. The tool is located under "Information" -> "View Response Headers". Also, the way I create my error pages is: I create some dummy error page. Let's say 403.php. I create an actual error page. For example ...


5

Use google sitemaps. We use them where I work, you can specify how often pages change, etc: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184&from=40318&rd=1 You cannot guarantee that they will crawl today, but for a small enough site, it will help them find your content faster.


5

Usually no but you have to be really sure that the URL is not present anywhere on the web, in your sitemap if you are publishing it. You also have to be careful on web server access statistics if you make them public. In addition You can always use the robot.txt file to tell Google not to crawl the URLs. But this is just security by obfuscation, if you ...


5

Yes, Google will find it somehow! They monitor people's browser/search history via Google Accounts/Toolbars/Social Networks and the like - then use that data to augment and prioritize their crawler. So if a user visits your page while logged into a google account with it's search history tracking enabled Google may find out about your page. You also ...


5

Unfortunately no one can say you when Google will. According to my experience it can take a lot of time, also several months.


5

There are many ways this can be done within Apache using Modules, or alternatively you can setup IP tables to do the job though personally I just use the modules. mod_security I've personally used this and it does the job well, a good article about limiting requests can be found here. mod_evasive Detection is performed by creating an internal dynamic ...


5

Yes. This is simple to demonstrate: A Google search for inurl:https yields 8.2 billion results.


5

Whether or not crawlers honor your robots.txt is entirely an on-your-honor based system. Nothing you put in that file is going to prevent a "fake" crawler from doing anything. With regards to User-agent:, that value is completely voluntary as well. You can instruct your browser, or any other HTTP client to send whatever value you want for that header.


5

Here is a robots.txt file that will allow Google, Bing, and Yahoo to crawl the site while disallowing all other crawling: User-Agent: * Disallow: / User-Agent: googlebot Disallow: User-Agent: bingbot Disallow: User-agent: slurp Disallow: Some crawlers ignore robots.txt entirely and crawl whatever they feel like. Some crawlers impersonate Googlebot or ...


5

Will google still crawl to Full_Res_Image and index that in web as well as google images? Yes. If it does crawl to the full res image, how do I then set alt tags on the full res image? You can't but you can still use other indicators of content such as: Use the alt attribute on the cropped image. By being the effective anchor text of that link ...


5

Most bots don't accept cookies (including Googlebot), however, some bots do. You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. ...


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


5

Google does appear to index XML sitemaps (like any XML file). It seems that if Google is aware of a URL and it returns a valid response then it's likely to get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. I believe the recommended method to prevent ...


5

Google does not index data URI images for Google image search. Google's John Mueller says so here and in the comments below. Because data URI images are not indexed in Google image search, the EXIF data in them is irrelevant. You can verify that these images are not indexed. I searched Google images for "data uri" and spot checked the results. All of ...


4

It's a bad idea to hide elements for google, google is smart and you could definitely get punished for this. Read more here: http://www.google.com/support/webmasters/bin/answer.py?answer=66355


4

I just realized that the user must have found an outbound link on this authenticated page, and then leaked the private URL as Referer when clicking through to some other website. This is the only possible explanation, and should really have been obvious from the start. Once leaked, the private URL may have been exposed to Google in a number of ways, e.g. ...


4

How about a video sitemap ? Read more here.


4

You can do this with section targeting: <!-- google_ad_section_start(weight=ignore) --> [stuff to ignore here] <!-- google_ad_section_end -->


4

Have you tried to view your site like a bot would? Xenu's Link Sleuth is a great way to look for bad links and should report links like you have above in its report. That should tell you where your problem lies.



Only top voted, non community-wiki answers of a minimum length are eligible