Tagged Questions

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
5answers
6k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
1
vote
1answer
164 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
8
votes
4answers
5k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
2
votes
1answer
60 views

Adding tagged / dynamic pages in sitemap

I've got a blog that's been running for about a year, with about 200 posts, and there should be about 220 pages to index (additional pages for about / contact etc). When I go to crawl the site I get ...
1
vote
2answers
205 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
2
votes
2answers
77 views

Is there a weight for html link?

I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google? 1. <a ...
3
votes
1answer
247 views

Is browser and bot whitelisting a practical approach?

With blacklisting it takes plenty of time to monitor events to uncover undesirable behavior and then taking corrective action. I would like to avoid that daily drudgery if possible. I'm thinking ...
8
votes
2answers
2k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
0
votes
3answers
113 views

Free standalone crawlers

Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors? Something that shows me all 404's, errors etc
4
votes
1answer
228 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
0
votes
1answer
587 views

Apache real time robot detection

Background Our website is being a. scraped - complete pages downloaded b. spidered - complete pages and sub links followed c. mined - specific files downloaded at high speed by hundreds of ...
1
vote
1answer
268 views

Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm ...
0
votes
1answer
79 views

How do I make the home page of the website to come up in the rankings than the internal pages? [duplicate]

Possible Duplicate: What are the best ways to increase your site's position in Google? Suppose I have a website, e.g. www.example.com that comes at number 6 on the Google search ...
0
votes
0answers
113 views

Disallow age gate/verification page in robots.txt?

Should I disallow the age gate page in robots.txt? I need to make sure the crawlers can get through the gate.
3
votes
2answers
188 views

Can Google tell the difference between a link in a blog post and a link in a blog comment?

When Google Blog Search crawls a page, can it tell whether a link is in the article or whether it's in a comment? If it can, does it grade the links, that is, a link from a blog comment is only going ...
1
vote
2answers
1k views

Does searchbots index content in iframe [duplicate]

Possible Duplicate: Do we still need to avoid using frame/iframe for good SEO? I have a html page that has a map of a country with lots of dots indicating location of companies through out ...
3
votes
1answer
923 views

Why is BingBot causing so many 404 errors by removing letters from url's?

Since the ForceRecrawl problem went away Bing has come back with some new trends. I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
216 views

Search engines crawling non-existent subdomains

A few years ago, a previous developer added my company's website to a link farm he ran. My company has a wildcard subdomain, so any subdomains will return the user to the homepage. As such, the dev ...
5
votes
1answer
241 views

What is User-Agent “AF_ID=<xxxxxx>”

We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like "AF_ID=". I've looked around for common ...
7
votes
2answers
679 views

@font-face, google fonts and SEO

Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
2
votes
1answer
574 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
1
vote
4answers
116 views

Is it bad for SEO to have a site with no click navigation?

I have a site where users can only get to most of the sites content through search. If I make a sitemap that has the URLs for all of the potential searches is it ok that these pages cannot be click ...
1
vote
1answer
542 views

Question about robots.txt Google shows urls are blocked

I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which ...
4
votes
2answers
693 views

How often are sitemap.xml checked for updates by crawlers?

How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
4
votes
2answers
538 views

Understanding Ajax crawling of search site

I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself. The base article explains the mechanism of making AJAX applications crawlable. All this stuff ...
2
votes
1answer
185 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
2
votes
1answer
274 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
0
votes
1answer
1k views

BingPreview user agent

I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b The ...
3
votes
1answer
375 views

How to interpret number of URL errors in Google webmaster tools

Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out ...
5
votes
1answer
468 views

What is the notified-CompliancePage page?

I sometimes have bots (I assume) hitting pages such as: /notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw== ...
6
votes
1answer
7k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
0
votes
1answer
252 views

Link error checking tool for linux [duplicate]

I use Xenu Kubj Sleuth to find out 404s and server errors on websites, but I'm not enjoying it, is there a better free tool out there that can preferably run in bash in linux that does the same thing ...
5
votes
3answers
522 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
2
votes
2answers
1k views

404 errors showing in Google Webmaster Tools

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found). But when I look at the source pages that are accused of ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
0
votes
0answers
56 views

Are there any good reasons to HTML5 for SEO at the moment? [duplicate]

Possible Duplicate: What are the SEO rules for HTML 5? Do crawlers ignore HTML5 tags (like <article>, <section>, <video>, <nav>, and so on...) when indexing the ...
2
votes
2answers
289 views

Tool to generate list of redirections (sitemap for .htaccess)

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site? I'm building new version of some site, but I don't want to get 404 errors for the old pages. ...
2
votes
1answer
2k views

how to make sure an individual website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
2answers
157 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
1
vote
1answer
337 views

Web crawler tool that generate a list of all the anchors used on a site

I'm looking for a web crawler tool that'll go through all the anchor text on a whole web site and generate a list for me to check. I want to make sure all the anchors are really descriptive outside ...
0
votes
2answers
115 views

404s on password protected content

I'm new to WordPress and SEO, generally, but we've been running into problems with our site that don't seem to make sense to me. The problem is that our editor likes to schedule posts and/or mark ...
4
votes
1answer
615 views

Is there any reason to allow Yahoo! Slurp to crawl my site?

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
5
votes
5answers
219 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...
-1
votes
1answer
171 views

Why my web site was visited by ARPA?

20 minutes ago a user-agent with 66.116.153.122 IP address has visited my web site. It's domain is rev.opentransfer.com.122.153.116.66.in-addr.arpa. User-agent: Mozilla/5.0 (Windows; U; Windows NT ...
5
votes
2answers
375 views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
1
vote
1answer
94 views

In User Agent strings, what does “AskTbTRL” denote?

Here are a bunch of examples: http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
2
votes
1answer
82 views

Does a page blocked for search engines get indexed after link share (+1)

I do my website development always on a subdomain, and that subdomain is blocked for searchengines. I do not want my lorem ipsum content and development domain name indexed by search engines. Let's ...
5
votes
2answers
634 views

How is crawler seeing unlinked directories / files?

I'm running a crawler on my website to test for broken links and such. It starts by using a URL like www.domain.com One curious thing is that it is showing directories with no internal links. For ...
6
votes
2answers
332 views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
1
vote
2answers
382 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?