A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
1answer
73 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
2answers
496 views

Fake links cause crawl error in Google Webmaster Tools

Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message: Google detected a significant increase in the number of URLs that return a 404 (Page ...
0
votes
1answer
116 views

Spiders crawling broken links

Tl:Dr Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin. The Problem It seems at least once a week we're getting a big ...
25
votes
1answer
5k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
1
vote
2answers
97 views

Can Google show some pages deleted six months back and blocked using Robot.txt?

If I have deleted some pages on my website six months back and blocked these pages using Robots.txt, can that non-existent page show up on Google?
2
votes
1answer
290 views

Is there a spider / link checker that can start deep inside a login-protected site

We use vendor hosted Blackboard for our distance education courses, but host course multimedia on our own servers. The multimedia server has been moved and the domain has changed. Blackboard DBAs have ...
1
vote
1answer
98 views

Crawling retail websites to use as templates

I'm setting up a commerce website. I really like the Overstock website, and I was wondering if could simply crawl the website for the webpages, remove Overstock's logos, and use their website as a ...
3
votes
2answers
616 views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
2
votes
1answer
180 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
8
votes
2answers
2k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
0
votes
2answers
388 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
3answers
2k views

Google Webmaster Tools - Crawl errors with 500 status code

I will try to explain the situation: In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs. After a while I changed something on my website, upgrading my pages ...
5
votes
4answers
285 views

Total Indexed 0, 99 not selected in webmaster tools

I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
0
votes
0answers
323 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
1
vote
2answers
166 views

Why is Verisign spidering my website?

In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
2
votes
1answer
84 views

Are the following assumptions about indexing and ranking correct?

Are the following statements true or false: If a page is crawled by Google, it automatically gets into the index (unless explicitly blocked by robots.txt) An indexed page always gets a ranking (high ...
2
votes
2answers
210 views

Can Google stop crawling/indexing your site because you are working a lot on it?

I think I read somewhere that Google could stop crawling/indexing a site/blog if there was a lot of work performed on it (like many pages are modified, deleted or loaded daily). It would wait until ...
3
votes
1answer
459 views

Preventing botvisits to website

Every time some user shares my website's address inside his/her tweets, the following bots come to my website: UnwindFetchor/1.0 (+http://www.gnip.com/) ShowyouBot (http://showyou.com/crawler) ...
2
votes
2answers
122 views

Prevent crawler that doesn't honoring robots.txt

I have some problem, when I try to write robots.txt for my site ... I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
2
votes
1answer
171 views

Googlebot keeps returning to crawl my page then the rankings fluctuate

I know that in order to get ranked the page has to be crawled by Google bots, I have had a page example of which I will get a better understanding of how the system works with Google. I have made a ...
1
vote
1answer
114 views

What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?

I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
5
votes
1answer
229 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
0
votes
1answer
130 views

403 Error Crawling Pages

I am receiving more than 6000 errors in Google Webmaster Tools. It is showing "access denied"; can anyone please help me out with resolving this?
1
vote
1answer
95 views

Recommend any tool to place .docx files on website for viewing so that search engines crawl it?

Recommend any tool to place .docx files on website for viewing so that search engines crawl it without any problem. I use scribd viewer, set the document as private on scribd and embed the file on my ...
1
vote
0answers
22 views

Does google crawl css scripts to check of hidden content [duplicate]

Possible Duplicate: hidden texts and page rank Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on ...
1
vote
1answer
141 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
2
votes
2answers
58 views

If C-Panel Indexing Manager sets a folder to “No Indexing” can it be crawled by a webcrawler?

People are able to view directories / folders on my site right now. So, they could go to mysite.com/images and see the full index. To prevent this, C-Panel offers an option to set a directory / ...
0
votes
0answers
17 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
0
votes
1answer
224 views

mod_rewrite and SEO friendliness

My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes. I have a unique, large PHP script, and I use the Apache ...
3
votes
1answer
191 views

webmaster tools 500 crawl error for asp faceted navigation that does not exist

I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results that can not be reached by a site visitor. These pages do not exist. We are using ...
2
votes
2answers
80 views

Google crawls my website too slowly [duplicate]

Possible Duplicate: What are the best ways to increase a site's position in Google? I think Google crawls it too slowly (every three days). I updated it daily and it's kind of ...
8
votes
1answer
214 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success ...
4
votes
2answers
518 views

Number of page requests by any Bot in 5 secs

I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?
2
votes
1answer
243 views

How to hide download file from bots? [duplicate]

Possible Duplicate: How to restrict the download of all files in a folder? I want to make a private file available for download but not use username/password protection. I want to put the ...
1
vote
1answer
110 views

Set initial robots.txt restrictions so high, Googlebot won't even try to access the updated less strict file.

When I initially setup my blog I put the restrictions on robots.txt way too high. They were so stick that googlebot can't even access the udated less strict robots file. I've been at this problem for ...
8
votes
1answer
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
3
votes
2answers
1k views

How to crawl a webPage with dynamic content added by javascript

I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using ...
2
votes
3answers
736 views

User-agent identification and SEO crawler database

I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values. I know there are 'trap' links that you can use to ...
3
votes
2answers
117 views

Yandex frequently replaces page names with ampersands

The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is: /mypage.aspx?param=value then it ...
3
votes
1answer
1k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
5answers
3k views

Why deny access to website for msnbot/bingbot?

I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
1
vote
5answers
6k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
1
vote
1answer
154 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
8
votes
4answers
5k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
2
votes
1answer
60 views

Adding tagged / dynamic pages in sitemap

I've got a blog that's been running for about a year, with about 200 posts, and there should be about 220 pages to index (additional pages for about / contact etc). When I go to crawl the site I get ...
1
vote
2answers
172 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
2
votes
2answers
76 views

Is there a weight for html link?

I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google? 1. <a ...
3
votes
1answer
242 views

Is browser and bot whitelisting a practical approach?

With blacklisting it takes plenty of time to monitor events to uncover undesirable behavior and then taking corrective action. I would like to avoid that daily drudgery if possible. I'm thinking ...
8
votes
2answers
2k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
0
votes
3answers
108 views

Free standalone crawlers

Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors? Something that shows me all 404's, errors etc