A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

3
votes
2answers
85 views

Am I being hacked or harvested? [closed]

I just did some log analysis of Q1 for one of my client's sites. There were about 30,000 sessions created for this site for each of the five separate IP addresses starting with 79.171.81.xxx. 150,000 ...
0
votes
2answers
107 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
1
vote
5answers
2k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
4
votes
3answers
206 views

Do i need to have a separate page for each article on my website?

I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google? Is is ...
0
votes
2answers
786 views

Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?

Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites. Do ...
3
votes
0answers
1k views

What is a good open source web crawler? [closed]

I'm looking for a good open source web crawler and i found these: DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
1
vote
1answer
99 views

SEO preparation for upcoming new site

I booked a domain and I created an image saying that the site is soon to be launched. What SEO preparations should I make to avoid harming my SEO ranking? Because my page is blank, the image itself ...
2
votes
1answer
104 views

google removed links from the index [duplicate]

I asked google to crawl a site through webmaster tool and include it to the index. How actually it was done: Firstly, i picked some urls manually. Approx. after half of a day those pages were ...
1
vote
1answer
80 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
2answers
522 views

Fake links cause crawl error in Google Webmaster Tools

Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message: Google detected a significant increase in the number of URLs that return a 404 (Page ...
0
votes
1answer
118 views

Spiders crawling broken links

Tl:Dr Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin. The Problem It seems at least once a week we're getting a big ...
25
votes
1answer
5k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
1
vote
2answers
97 views

Can Google show some pages deleted six months back and blocked using Robot.txt?

If I have deleted some pages on my website six months back and blocked these pages using Robots.txt, can that non-existent page show up on Google?
2
votes
1answer
305 views

Is there a spider / link checker that can start deep inside a login-protected site

We use vendor hosted Blackboard for our distance education courses, but host course multimedia on our own servers. The multimedia server has been moved and the domain has changed. Blackboard DBAs have ...
1
vote
1answer
98 views

Crawling retail websites to use as templates

I'm setting up a commerce website. I really like the Overstock website, and I was wondering if could simply crawl the website for the webpages, remove Overstock's logos, and use their website as a ...
3
votes
2answers
661 views

How can I prevent Googlebot from indexing web service URL's?

I have a problem with googlebot and bingbot accessing our web service. We have a search application created in aspx that is integrated into a drupal website. Users perform searches based on postal ...
2
votes
1answer
181 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
8
votes
2answers
2k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
0
votes
2answers
420 views

GWT: reporting crawling errors for non existing links

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
3answers
2k views

Google Webmaster Tools - Crawl errors with 500 status code

I will try to explain the situation: In December 2012 I deployed a website with a sitemap.xml, etc... there were circa 2000 URLs. After a while I changed something on my website, upgrading my pages ...
5
votes
4answers
294 views

Total Indexed 0, 99 not selected in webmaster tools

I recently changed domain names from xxx.com to xxx.ca I created all the 301's submitted sitemaps and submitted a change of address to google webmaster tools on Dec 16th. I've since seen urls change ...
0
votes
0answers
336 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
1
vote
2answers
189 views

Why is Verisign spidering my website?

In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
2
votes
1answer
84 views

Are the following assumptions about indexing and ranking correct?

Are the following statements true or false: If a page is crawled by Google, it automatically gets into the index (unless explicitly blocked by robots.txt) An indexed page always gets a ranking (high ...
2
votes
2answers
212 views

Can Google stop crawling/indexing your site because you are working a lot on it?

I think I read somewhere that Google could stop crawling/indexing a site/blog if there was a lot of work performed on it (like many pages are modified, deleted or loaded daily). It would wait until ...
3
votes
1answer
494 views

Preventing botvisits to website

Every time some user shares my website's address inside his/her tweets, the following bots come to my website: UnwindFetchor/1.0 (+http://www.gnip.com/) ShowyouBot (http://showyou.com/crawler) ...
2
votes
2answers
124 views

Prevent crawler that doesn't honoring robots.txt

I have some problem, when I try to write robots.txt for my site ... I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
2
votes
1answer
171 views

Googlebot keeps returning to crawl my page then the rankings fluctuate

I know that in order to get ranked the page has to be crawled by Google bots, I have had a page example of which I will get a better understanding of how the system works with Google. I have made a ...
1
vote
1answer
119 views

What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?

I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
5
votes
1answer
246 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
0
votes
1answer
136 views

403 Error Crawling Pages

I am receiving more than 6000 errors in Google Webmaster Tools. It is showing "access denied"; can anyone please help me out with resolving this?
1
vote
1answer
99 views

Recommend any tool to place .docx files on website for viewing so that search engines crawl it?

Recommend any tool to place .docx files on website for viewing so that search engines crawl it without any problem. I use scribd viewer, set the document as private on scribd and embed the file on my ...
1
vote
0answers
22 views

Does google crawl css scripts to check of hidden content [duplicate]

Possible Duplicate: hidden texts and page rank Does googlebot crawl CSS scripts to determine if content is hidden or not? If so, is it advisable to hide or not to hide content to improve on ...
1
vote
1answer
143 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
2
votes
2answers
58 views

If C-Panel Indexing Manager sets a folder to “No Indexing” can it be crawled by a webcrawler?

People are able to view directories / folders on my site right now. So, they could go to mysite.com/images and see the full index. To prevent this, C-Panel offers an option to set a directory / ...
0
votes
0answers
17 views

Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]

Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
0
votes
1answer
245 views

mod_rewrite and SEO friendliness

My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes. I have a unique, large PHP script, and I use the Apache ...
3
votes
1answer
198 views

webmaster tools 500 crawl error for asp faceted navigation that does not exist

I am getting around 2,500 type 500 URL errors in Google webmaster tools. These pages are faceted navigation results that can not be reached by a site visitor. These pages do not exist. We are using ...
2
votes
2answers
80 views

Google crawls my website too slowly [duplicate]

Possible Duplicate: What are the best ways to increase a site's position in Google? I think Google crawls it too slowly (every three days). I updated it daily and it's kind of ...
8
votes
1answer
223 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success ...
4
votes
2answers
568 views

Number of page requests by any Bot in 5 secs

I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?
2
votes
1answer
257 views

How to hide download file from bots? [duplicate]

Possible Duplicate: How to restrict the download of all files in a folder? I want to make a private file available for download but not use username/password protection. I want to put the ...
1
vote
1answer
111 views

Set initial robots.txt restrictions so high, Googlebot won't even try to access the updated less strict file.

When I initially setup my blog I put the restrictions on robots.txt way too high. They were so stick that googlebot can't even access the udated less strict robots file. I've been at this problem for ...
8
votes
1answer
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
3
votes
2answers
1k views

How to crawl a webPage with dynamic content added by javascript

I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using ...
2
votes
3answers
777 views

User-agent identification and SEO crawler database

I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values. I know there are 'trap' links that you can use to ...
3
votes
2answers
117 views

Yandex frequently replaces page names with ampersands

The Yandex spider is a frequent visitor to one of the sites I manage. On occasion it replaces the page name with two ampersands and a space. So if the page is: /mypage.aspx?param=value then it ...
3
votes
1answer
1k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
3
votes
5answers
3k views

Why deny access to website for msnbot/bingbot?

I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
1
vote
5answers
6k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.