A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

8
votes
6answers
8k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
3
votes
2answers
83 views

How to set crawl rate in Google Webmaster Tools for seven days or a month?

My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
2
votes
1answer
530 views

Do all search engines obey nofollow rel attribute on hyperlinks

I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value. Is this the case. From my research Google, Yahoo and Bing seem to obey this attribute ...
2
votes
2answers
809 views

All the IPs of search engine bots?

if I want to filter out all the search engine bots for example for a script that counts users that are online is there website or a list which is allways up to date and includes the ip addresses of ...
1
vote
2answers
801 views

SEO: disallowing Google from indexing forms?

From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases. So I'm wondering what's the point of letting ...
1
vote
2answers
55 views

Crawling web to build a directory of a niche industry for SEO

I heard that some web sites crawl web to build a directory of companies in a niche domain. So, when people search for the name of the company they find that directory page. I wonder if this is ...
2
votes
1answer
147 views

What is user agent Issbot

I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
3
votes
3answers
362 views

Is my robots.txt working as it should?

I want crawlers to have access to http://www.example.com but not http://www.example.com/ My robots.txt is as follows: User-agent: * Allow: /$ Disallow: / My site is in Google search results, but I ...
0
votes
2answers
132 views

Can internal linking create excessive web server load?

SE sites allow tags chaining like this: /questions/tagged/seo+google+search-engines+research+keywords This site has more than 700 tags. If up to 5 tags can be chained, there are 700*700*700*700*...
1
vote
2answers
115 views

SEO - Crawl Errors

I've been optimizing my website for search engines, and for some reason I have a crawl error on a page that has not been found at 'www.peach-designs.com/a' which is not a page, is not a link, or is ...
1
vote
1answer
610 views

How does bing-bot( is that the right spider-name? ) and googlebot interpret 301 redirect?

I've been looking for documentation on how the Microsoft and Google bots interpret 301 redirects. It seems that google-bot stores documents on a url based index system. But I haven't been able to ...
1
vote
2answers
207 views

Adult website crawlability issue

I got a mission and asking for some help with this scenario for a new website. The site is an adult escorts advertising service. It is developed fine, works and looks good, but the issue is with ...
4
votes
5answers
1k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
2answers
330 views

If an visitors IP address contains “google” or a similar keyword, does this mean they were a crawler?

I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, ...
0
votes
2answers
282 views

Menu hidden links and bots!

I want to make a menu with a list with hidden links and when the user points the menu with he mouse the links will appear, my problem is that the bots may think this as a deceiving tactic and lower my ...
8
votes
0answers
521 views

Is there a search engine that indexes source code of a web-page? [closed]

I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of ...
2
votes
2answers
3k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
1answer
253 views

Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?

We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us. However, in addition to the hierarchical page ...
2
votes
3answers
187 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
10
votes
1answer
684 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
12
votes
3answers
5k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
6
votes
1answer
249 views

List of all URI elements for a website

Does anyone know a good way of getting a list of all the URI elements for a website? I plan on moving a website to a new CMS and would like to setup some 301 redirects for articles, images, css, and ...
17
votes
4answers
4k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
2
votes
1answer
156 views

What's the extent of the “damage” to having a broken URL found by the spiders (showing in Google Webmaster Tools)?

An error with our URL rewrite rules caused some pages to be not found (404 errors) when Google crawled our site. I regularly watch for errors and fix database or coding issues when they arise. We ...
0
votes
2answers
204 views

Where can I host this?

I'm looking for a VPS or dedicated that can have a web crawler running all day long. It would gather data and then add it to some websites. These websites currently have around 1k unique visits/day, ...
1
vote
1answer
89 views

What are Sitemap content tags?

i have recently generated sitemap for google. I included following loc, lastmod, changefreq. My site is in two languages. Do i need to add some other tags for specifying languages too
4
votes
1answer
436 views

Should I block search engines with robots.txt on my Facebook app?

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
6
votes
2answers
234 views

Does the adsense bot ever get bored?

Each time someone clicks on a page like http://www.example.com/crazy.cgi?input=silly+banana where the "silly banana" bit represents some arbitrary user input which changes each time, I get yet ...
4
votes
2answers
6k views

Sosospider: what does it actually want?

On one website I have, looking at the logs, I find lots and lots of "Sosospider" hits. Is this the same for everyone, or just me? Now, I have never once been sent any traffic from anything which looks ...
3
votes
2answers
5k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
13
votes
3answers
2k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and i wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
0
votes
1answer
61 views

Possible reasons RSS generated content won't index

I built a digg-like clone at fantasybookmark.com. Google crawled the site, but only indexed about 21 pages out of the 2,000 or so (and growing) pages on the site. We use rss feeds to find and ...
3
votes
1answer
153 views

Is there a good way to filter all the robots and domains they are on?

I have been working more on the bot filter for my website, but by no means is it complete. So far I have the main ones: Google, Yahoo, MSN, Baidu, Amazon, and a few others... Right now, I am ...
1
vote
1answer
114 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
12
votes
4answers
484 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
2
votes
3answers
622 views

Blocking all search engines except the big ones

I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and ...
2
votes
2answers
1k views

How to detect search engine bots reliably within asp.net website?

What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user ...
24
votes
4answers
947 views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
9
votes
8answers
7k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
7
votes
7answers
3k views

Is there an index of the IP addresses used by indexing bots?

I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list. Is ...
20
votes
6answers
3k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...