Tag Info

Hot answers tagged

16

Apparently it does, some, according to Matt Cutts: "For a while, we were scanning within JavaScript, and we were looking for links. Google has gotten smarter about JavaScript and can execute some JavaScript. I wouldn't say that we execute all JavaScript, so there are some conditions in which we don't execute JavaScript. Certainly there are some common, ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


12

This is Vincent from Bing Webmaster Tools. This was a Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin (Beta) for IIS and Apache - see the following for more details: Bing Sitemap Plugin (Beta) - Bing Webmaster Tools Help & How-To Center Building Sitemaps Manually?… Stop Until You Read This! - Bing Webmaster Blog You ...


11

Yes and no, he's conflating two different things. In late 90's/early 2000's, when designers needed to use fonts for design reasons, they embedded graphics files in HTML pages. Search engines cannot understand these, nor can the screen readers used by blind and partially sighted people. This is because the only HTML code in the page for a menu link, for ...


8

I'm the primary designer and author of a fairly large-scale web crawler (see http://metadatalabs.com/mlbot). What you're asking touches on a topic that's very important to us--perhaps the most important part of running a crawler: that of politeness. First: the reason for the "Mozilla" thing is to tell the site what your browser capabilities are. If your ...


8

I would see no reason to unblock Yahoo's search bot. There are 2 ways Yahoo could be using the Bing Engine. They could integrate it with their own and set a hybrid crawler loose on the net or they could just be sending the queries into the Bing Engine and spitting out the results they get back. From the document Joel linked to there is another document ...


8

Can they see it? Yes. Can they find it? Not without help. Web crawlers typically find pages to crawl by following links to them on other pages. Some crawlers (e.g. search engine crawlers) will also crawl pages listed in special XML files. So if there is no link to page on your website or any other website then that page will not be crawled (pages that ...


8

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


7

It appears that they've just started rolling out Bing search results on Yahoo! for about 25% of the searches, with the goal of completely switching over in August/September. I would bet that the Yahoo spider isn't even running any more and if it is, it's not very relevant.


7

I'm glad you know there isn't going to be a bulletproof way to accomplish this. That means your outlook is at least realistic. Since JavaScript is not an option I would say you're left with: Check the user-agent for the word "bot" in it. That will catch most of them. Compile a list of known bots and filter them based on some kind of unique identifier, ...


7

It would seem that Google has probably not yet updated it's cache of your robots.txt file. Your current robots.txt file (above) does not look as if it should be blocking your sitemap URL. I guess google just hasnt updated its cache. There is no need to guess. In Google Webmaster Tools (GWT) under "Health" > "Blocked URLs", you can see when your ...


6

If you are using windows 7 the best tool is IIS7's SEO Toolkit 1.0. It is free and you can download it for free. The tool will scan any site and tell you where all of the dead links are, what pages take to long to load, what pages have missing titles, duplicate titles, same for keywords and descriptions, and what pages have broken HTML.


6

If it's a HTML document, include the following meta tag in it: <meta name="robots" content="noindex" /> This will tell Google and every other search engine that understands the tag (which should be pretty much all of them, since it's been around since 1996) not to index the page. For non-HTML documents, you can configure your webserver to send the ...


6

I don't think people should ban bing bot. Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/ where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video: http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jz Googlebot is just as notorious in excessive ...


6

You can block bots but it depends on what you want for your website. You can block search engine bots if you don't want to see your website indexed in a particular search engine. Example: Yandex is russian search engine. You can block its bot if your business is not targeting Russia. You can block SEO bots if you don't want to use their web analytics ...


6

Google has no knowledge of the actual file structure of your web app and what is accessible to the public. A news site, for example, will have hundreds of articles to view, but might only have one actual script: article.php. Server-side directives might allow pretty canonical URLs like /category/283423-pretty-name to point to /article.php, and this is what ...


6

This is no doubt dependent on your remaining 200+ lines, but what you have posted so far would seem to be reducible to just 2 lines: RewriteRule ^patients/billing/(FAQ_billing|getintouch).html$ $1.php [L,NC] RewriteRule ^patients/findadoctor/([a-z]).html$ findadoctor.php?id=$1 [L,NC] This shouldn't be reducing your site to a crawl. As mentioned in ...


6

For years, I haven't seen a site ranking in Google that doesn't have a domain name. That doesn't mean it isn't possible, but few (if any) sites are doing that now. Here are a bunch of reasons that you should get a domain name: IP addresses cannot be moved from one web host to another the way that domain names can. IP Addresses change. It is very ...


5

Mozilla/2.0 and Mozilla/5.0 are both references to the Mozilla browser. It has become largely meaningless, with many crawlers using it, but should tell the site to treat your crawler as it would any random user browsing with a regular browser. It is however good etiquette to include an URL linking to a page about who you are and why you are crawling in the ...


5

The most probable reason is that the pages won't return a 403 header. You can check that using the Web Developer Toolbar in Firefox or Chrome. The tool is located under "Information" -> "View Response Headers". Also, the way I create my error pages is: I create some dummy error page. Let's say 403.php. I create an actual error page. For example ...


5

It looks like it is just a search engine crawler for a website in China. The lack of traffic may have to do with the fact that your site is in English (assumption) and probably isn't what its users are looking for. I would think the crawler wouldn't crawl you since your site is not in Chinese (assumption) but maybe their system isn't that sophisticated or ...


5

There are three fairly simple ways: Use Google Analytics, which will process and handle all the data for you, and present you with detailed statistics for visitors and how they got to your site. This is by far the easiest solution. Use Javascript to do the counting. When the page has loaded, generate an AJAX request to your counting script. Robots and ...


5

Use google sitemaps. We use them where I work, you can specify how often pages change, etc: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184&from=40318&rd=1 You cannot guarantee that they will crawl today, but for a small enough site, it will help them find your content faster.


5

Yes. You can disallow everything first, then allow the folders you want to be indexed. User-agent: * Disallow: / Allow: /index.html Allow: /example/ Allow: /example2/ Allow: /example3/ The reason this works is that Google (and Bing) follow CSS-style specificity rules when reading robots.txt files, where more specific rules take precedence over less ...


5

Usually no but you have to be really sure that the URL is not present anywhere on the web, in your sitemap if you are publishing it. You also have to be careful on web server access statistics if you make them public. In addition You can always use the robot.txt file to tell Google not to crawl the URLs. But this is just security by obfuscation, if you ...


5

Yes, Google will find it somehow! They monitor people's browser/search history via Google Accounts/Toolbars/Social Networks and the like - then use that data to augment and prioritize their crawler. So if a user visits your page while logged into a google account with it's search history tracking enabled Google may find out about your page. You also ...


5

It doesn't sound like something you should worry about too much, google crawls some of my sites in a similar way, but it could be a fake bot, so check out the links below... The Baidu Spider FAQ's (Chrome's built in translation) have this to say:- 4. Why Baiduspider crawls my site continuously? In order to ensure the latest information is presented, ...


5

you might be better off using your firewall instead of relying on a php script, since after all the php script will only affect php page loads and not static files like images. Plus, having php log all connections and check for >800 from the same ip for every request is going to add some serious overhead to your server, maybe even more so than the requests ...


5

There are many ways this can be done within Apache using Modules, or alternatively you can setup IP tables to do the job though personally I just use the modules. mod_security I've personally used this and it does the job well, a good article about limiting requests can be found here. mod_evasive Detection is performed by creating an internal dynamic ...


5

Great question, and one many webmasters might be interested in since the Baidu spider is notoriously aggressive and can zap resources from servers... As indicated in Baidu's Web Search news, the Baidu spider does not support the Crawl-delay notification setting, and instead requires you to register and verify your site with its Baidu Webmaster Tools ...



Only top voted, non community-wiki answers of a minimum length are eligible