Tagged Questions

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

2
votes
1answer
263 views

How to hide download file from bots? [duplicate]

Possible Duplicate: How to restrict the download of all files in a folder? I want to make a private file available for download but not use username/password protection. I want to put the ...
1
vote
1answer
112 views

Set initial robots.txt restrictions so high, Googlebot won't even try to access the updated less strict file.

When I initially setup my blog I put the restrictions on robots.txt way too high. They were so stick that googlebot can't even access the udated less strict robots file. I've been at this problem for ...
8
votes
1answer
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
3
votes
2answers
1k views

How to crawl a webPage with dynamic content added by javascript

I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using ...
2
votes
3answers
800 views

User-agent identification and SEO crawler database

I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values. I know there are 'trap' links that you can use to ...
3
votes
2answers
117 views

Yandex frequently replaces page names with ampersands

The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is: /mypage.aspx?param=value then it ...
3
votes
1answer
1k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
5answers
3k views

Why deny access to website for msnbot/bingbot?

I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
1
vote
5answers
6k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
1
vote
1answer
167 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
8
votes
4answers
5k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
2
votes
1answer
60 views

Adding tagged / dynamic pages in sitemap

I've got a blog that's been running for about a year, with about 200 posts, and there should be about 220 pages to index (additional pages for about / contact etc). When I go to crawl the site I get ...
1
vote
2answers
207 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
2
votes
2answers
77 views

Is there a weight for html link?

I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google? 1. <a ...
3
votes
1answer
250 views

Is browser and bot whitelisting a practical approach?

With blacklisting it takes plenty of time to monitor events to uncover undesirable behavior and then taking corrective action. I would like to avoid that daily drudgery if possible. I'm thinking ...
8
votes
2answers
2k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
0
votes
3answers
114 views

Free standalone crawlers

Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors? Something that shows me all 404's, errors etc
4
votes
1answer
228 views

Using rel=canonical and noindex in a 1-n partners environment

We sell complete sites including the domains to several partners that create content that is shown together at the main site. What we want to achieve is that the main site copy is the original but the ...
0
votes
1answer
599 views

Apache real time robot detection

Background Our website is being a. scraped - complete pages downloaded b. spidered - complete pages and sub links followed c. mined - specific files downloaded at high speed by hundreds of ...
1
vote
1answer
268 views

Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm ...
0
votes
1answer
79 views

How do I make the home page of the website to come up in the rankings than the internal pages? [duplicate]

Possible Duplicate: What are the best ways to increase your site's position in Google? Suppose I have a website, e.g. www.example.com that comes at number 6 on the Google search ...
0
votes
0answers
117 views

Disallow age gate/verification page in robots.txt?

Should I disallow the age gate page in robots.txt? I need to make sure the crawlers can get through the gate.
3
votes
2answers
188 views

Can Google tell the difference between a link in a blog post and a link in a blog comment?

When Google Blog Search crawls a page, can it tell whether a link is in the article or whether it's in a comment? If it can, does it grade the links, that is, a link from a blog comment is only going ...
1
vote
2answers
1k views

Does searchbots index content in iframe [duplicate]

Possible Duplicate: Do we still need to avoid using frame/iframe for good SEO? I have a html page that has a map of a country with lots of dots indicating location of companies through out ...
3
votes
1answer
942 views

Why is BingBot causing so many 404 errors by removing letters from url's?

Since the ForceRecrawl problem went away Bing has come back with some new trends. I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
217 views

Search engines crawling non-existent subdomains

A few years ago, a previous developer added my company's website to a link farm he ran. My company has a wildcard subdomain, so any subdomains will return the user to the homepage. As such, the dev ...
5
votes
1answer
242 views

What is User-Agent “AF_ID=<xxxxxx>”

We periodically (more frequently recently) have very agressive crawling activity coming from EC2 instances that give us a user agent that looks like "AF_ID=". I've looked around for common ...
7
votes
2answers
686 views

@font-face, google fonts and SEO

Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
2
votes
1answer
576 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
1
vote
4answers
116 views

Is it bad for SEO to have a site with no click navigation?

I have a site where users can only get to most of the sites content through search. If I make a sitemap that has the URLs for all of the potential searches is it ok that these pages cannot be click ...
1
vote
1answer
543 views

Question about robots.txt Google shows urls are blocked

I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which ...
4
votes
2answers
705 views

How often are sitemap.xml checked for updates by crawlers?

How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
4
votes
2answers
540 views

Understanding Ajax crawling of search site

I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself. The base article explains the mechanism of making AJAX applications crawlable. All this stuff ...
2
votes
1answer
185 views

Will google “forget” unlinked pages?

If i remove all links to a page, but do not delete the page from the site (nor block it from being requested), will google eventually "forget" about it when it reindexes the site? Assuming of course ...
2
votes
1answer
275 views

Stop bots from crawling old links with extensions

I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist. So I'm trying to find out if there is a ...
0
votes
1answer
1k views

BingPreview user agent

I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b The ...
3
votes
1answer
376 views

How to interpret number of URL errors in Google webmaster tools

Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out ...
5
votes
1answer
474 views

What is the notified-CompliancePage page?

I sometimes have bots (I assume) hitting pages such as: /notified-CompliancePage?aHR0cDovL3d3dy5teXBpdm90cy5jb20vZGFpbHlub3Rlcw== ...
6
votes
1answer
7k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
0
votes
1answer
252 views

Link error checking tool for linux [duplicate]

I use Xenu Kubj Sleuth to find out 404s and server errors on websites, but I'm not enjoying it, is there a better free tool out there that can preferably run in bash in linux that does the same thing ...
5
votes
3answers
525 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
2
votes
2answers
1k views

404 errors showing in Google Webmaster Tools

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found). But when I look at the source pages that are accused of ...
3
votes
1answer
1k views

Is feeding Googlebot and Bingbot crawlers with “special” data common thing in rich internet applications?

So I faced next problem: my server and my rich internet application (RIA) client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull ...
0
votes
0answers
56 views

Are there any good reasons to HTML5 for SEO at the moment? [duplicate]

Possible Duplicate: What are the SEO rules for HTML 5? Do crawlers ignore HTML5 tags (like <article>, <section>, <video>, <nav>, and so on...) when indexing the ...
2
votes
2answers
290 views

Tool to generate list of redirections (sitemap for .htaccess)

Is there any good tool (preferably for Ubuntu), that can help me generate a list of URLs used on a site? I'm building new version of some site, but I don't want to get 404 errors for the old pages. ...
2
votes
1answer
2k views

how to make sure an individual website is not indexed by search engines?

I'm setting up a personal website with a blog that I would like to be indexed by search engines. But the blog will link to CV with personal information that shouldn't be indexed. Can it be reliably ...
2
votes
2answers
158 views

Does Yahoo Slurp crawl hammer a site?

A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
1
vote
1answer
338 views

Web crawler tool that generate a list of all the anchors used on a site

I'm looking for a web crawler tool that'll go through all the anchor text on a whole web site and generate a list for me to check. I want to make sure all the anchors are really descriptive outside ...
0
votes
2answers
115 views

404s on password protected content

I'm new to WordPress and SEO, generally, but we've been running into problems with our site that don't seem to make sense to me. The problem is that our editor likes to schedule posts and/or mark ...
4
votes
1answer
638 views

Is there any reason to allow Yahoo! Slurp to crawl my site?

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...