A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

4
votes
5answers
1k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
2answers
329 views

If an visitors IP address contains “google” or a similar keyword, does this mean they were a crawler?

I have a huge list of IP addresses recorded from various visitors to a website. A huge amount of the visitors, in some months over 70%, came from IP addresses that contained keywords such as google, ...
0
votes
2answers
282 views

Menu hidden links and bots!

I want to make a menu with a list with hidden links and when the user points the menu with he mouse the links will appear, my problem is that the bots may think this as a deceiving tactic and lower my ...
8
votes
0answers
521 views

Is there a search engine that indexes source code of a web-page? [closed]

I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of ...
2
votes
2answers
3k views

Robots.txt and Regular Expressions

If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
1answer
253 views

Should I strategically use NoFollow on internal links to make our indexing crawl more efficient?

We have a well designed internal linking system that makes certain no page is more than four pages deep, and the indexing process has gone well for us. However, in addition to the hierarchical page ...
2
votes
3answers
186 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
10
votes
1answer
682 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
11
votes
3answers
5k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
6
votes
1answer
249 views

List of all URI elements for a website

Does anyone know a good way of getting a list of all the URI elements for a website? I plan on moving a website to a new CMS and would like to setup some 301 redirects for articles, images, css, and ...
17
votes
4answers
4k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
2
votes
1answer
156 views

What's the extent of the “damage” to having a broken URL found by the spiders (showing in Google Webmaster Tools)?

An error with our URL rewrite rules caused some pages to be not found (404 errors) when Google crawled our site. I regularly watch for errors and fix database or coding issues when they arise. We ...
0
votes
2answers
202 views

Where can I host this?

I'm looking for a VPS or dedicated that can have a web crawler running all day long. It would gather data and then add it to some websites. These websites currently have around 1k unique visits/day, ...
1
vote
1answer
88 views

What are Sitemap content tags?

i have recently generated sitemap for google. I included following loc, lastmod, changefreq. My site is in two languages. Do i need to add some other tags for specifying languages too
4
votes
1answer
436 views

Should I block search engines with robots.txt on my Facebook app?

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
6
votes
2answers
233 views

Does the adsense bot ever get bored?

Each time someone clicks on a page like http://www.example.com/crazy.cgi?input=silly+banana where the "silly banana" bit represents some arbitrary user input which changes each time, I get yet ...
4
votes
2answers
6k views

Sosospider: what does it actually want?

On one website I have, looking at the logs, I find lots and lots of "Sosospider" hits. Is this the same for everyone, or just me? Now, I have never once been sent any traffic from anything which looks ...
3
votes
2answers
5k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
0
votes
1answer
61 views

Possible reasons RSS generated content won't index

I built a digg-like clone at fantasybookmark.com. Google crawled the site, but only indexed about 21 pages out of the 2,000 or so (and growing) pages on the site. We use rss feeds to find and ...
3
votes
1answer
153 views

Is there a good way to filter all the robots and domains they are on?

I have been working more on the bot filter for my website, but by no means is it complete. So far I have the main ones: Google, Yahoo, MSN, Baidu, Amazon, and a few others... Right now, I am ...
1
vote
1answer
114 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
12
votes
4answers
482 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
2
votes
3answers
619 views

Blocking all search engines except the big ones

I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and ...
2
votes
2answers
1k views

How to detect search engine bots reliably within asp.net website?

What is the best way to detect bot generated activity (clicks/page visits) etc on a asp.net website? We have a website where we track leads generated to external websites, we track the IP of the user ...
24
votes
4answers
939 views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
9
votes
8answers
6k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
7
votes
7answers
3k views

Is there an index of the IP addresses used by indexing bots?

I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list. Is ...
20
votes
6answers
3k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...