The web-crawlers tag has no wiki summary.
2
votes
1answer
43 views
How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
3
votes
2answers
121 views
Can the .htaccess file slow down a website to a crawl? If so, are there better ways to solve these problems with different rewrite rules and such?
here is my htaccess file......
RewriteCond %{REQUEST_URI} ^/patients/billing/FAQ_billing\.html$ [OR]
RewriteCond %{REQUEST_URI} ^/patients/billing/getintouch\.html$
RewriteRule ...
0
votes
0answers
15 views
SEO - Big revamp of my URL structure - please suggestions [duplicate]
I have a big network of sites in many countries. There are over 30-40 Million indexed pages in 34 countries, what will be re-indexed hopefully and will be redirected with 301 to the new URL structure, ...
0
votes
1answer
41 views
Google adsense crawler unable to access your page [closed]
I have a problem which I have been trying to solve by Googling it but it seems I can’t get no answer. I have been using plain PHP and HTML on my site and everything was ok concerning my Google ...
0
votes
2answers
53 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
0
votes
3answers
166 views
Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?
Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
3
votes
0answers
352 views
What is a good open source web crawler? [closed]
I'm looking for a good open source web crawler and i found these:
DataparkSearch, GNU Wget, GRUB, Heritrix, ht://Dig, HTTrack, ICDL, mnoGoSearch, Nutch, Open Search Server, PHP-Crawler, tkWWW Robot, ...
1
vote
1answer
81 views
SEO preparation for upcoming new site
I booked a domain and I created an image saying that the site is soon to be launched.
What SEO preparations should I make to avoid harming my SEO ranking? Because my page is blank, the image itself ...
22
votes
1answer
3k views
Bingbot request for trafficbasedsspsitemap.xml which does not exist
The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are
Path: /trafficbasedsspsitemap.xml
Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
2
votes
1answer
77 views
Is there a spider / link checker that can start deep inside a login-protected site
We use vendor hosted Blackboard for our distance education courses, but host course multimedia on our own servers. The multimedia server has been moved and the domain has changed. Blackboard DBAs have ...
1
vote
1answer
62 views
Crawling retail websites to use as templates
I'm setting up a commerce website. I really like the Overstock website, and I was wondering if could simply crawl the website for the webpages, remove Overstock's logos, and use their website as a ...
3
votes
1answer
157 views
Preventing botvisits to website
Every time some user shares my website's address inside his/her tweets, the following bots come to my website:
UnwindFetchor/1.0 (+http://www.gnip.com/)
ShowyouBot (http://showyou.com/crawler)
...
2
votes
2answers
87 views
Prevent crawler that doesn't honoring robots.txt
I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
2
votes
1answer
25 views
If C-Panel Indexing Manager sets a folder to “No Indexing” can it be crawled by a webcrawler?
People are able to view directories / folders on my site right now. So, they could go to mysite.com/images and see the full index. To prevent this, C-Panel offers an option to set a directory / ...
0
votes
0answers
17 views
Is this Anti-Scraping technique viable with Crawl-Delay? [duplicate]
Possible Duplicate:
How do spambots work?
I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...
8
votes
1answer
154 views
Disqus thread migration. Gotchas?
I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads.
I've had partial success ...
2
votes
3answers
309 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
3
votes
4answers
1k views
Why deny access to website for msnbot/bingbot?
I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
2
votes
2answers
72 views
Is there a weight for html link?
I would like to know if there is any weight associated with a html link (its backlink) when google does crawling/indexing. Will 1,2 & 3 be ever considered as a backlink by Google?
1. <a ...
0
votes
3answers
73 views
Free standalone crawlers
Are there any free standalone crawlers that I can use apart from Google webmaster to analyze errors?
Something that shows me all 404's, errors etc
0
votes
0answers
42 views
Is there an automated way to take site inventory?
Is there a way to take site inventory using a crawler program that checks either the sources of images for specific servers that serve ads, or, that the crawler looks at a page for specific (html5?) ...
0
votes
1answer
72 views
How do I make the home page of the website to come up in the rankings than the internal pages? [duplicate]
Possible Duplicate:
What are the best ways to increase your site's position in Google?
Suppose I have a website, e.g. www.example.com that comes at number 6 on the Google search ...
0
votes
0answers
48 views
Need to know how to configure robots.txt for an age gate/age verification website
is it the normal allow all in the file? I need to make sure the crawlers can get through the gate. Thanks!
3
votes
1answer
481 views
Why is BingBot causing so many 404 errors by removing letters from url's?
Since the ForceRecrawl problem went away Bing has come back with some new trends.
I'm seeing many URLs missing the last letter, or missing a few letters, and some other wired guess URLs. It also ...
1
vote
1answer
174 views
Search engines crawling non-existent subdomains
A few years ago, a previous developer added my company's website to a link farm he ran. My company has a wildcard subdomain, so any subdomains will return the user to the homepage. As such, the dev ...
3
votes
2answers
257 views
How often are sitemap.xml checked for updates by crawlers?
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
2
votes
1answer
205 views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
2
votes
0answers
916 views
Is feeding Googlebot and Bingbot crawlers with “special” data common thing in RIAs?
So I faced next problem: my server and my RIA client are using/generating so much JSON data that is rendered on fly even on just page creation complete that standard dull no js crawler will see not ...
-1
votes
1answer
137 views
Why my web site was visited by ARPA?
20 minutes ago a user-agent with 66.116.153.122 IP address has visited my web site. It's domain is rev.opentransfer.com.122.153.116.66.in-addr.arpa. User-agent: Mozilla/5.0 (Windows; U; Windows NT ...
5
votes
2answers
316 views
How is crawler seeing unlinked directories / files?
I'm running a crawler on my website to test for broken links and such.
It starts by using a URL like www.domain.com
One curious thing is that it is showing directories with no internal links. For ...
4
votes
2answers
201 views
Is it possible for web crawlers to see static pages without following a link to them?
If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
2
votes
2answers
141 views
How can I archive (for historical record) my website?
If I launch a website that displays 1000's of pages of content (text) delivered dynamically from data in a database and and I want some sort of record to prove that this data had been published (on ...
0
votes
1answer
35 views
Google spidering: make content available but unsuable
I'm trying to figure out how to make it so that my member based social network site can be spidered but it not possible for guests to post or interact with content in any way. What is the best way to ...
1
vote
1answer
405 views
Robots.txt practices with .htaccess redirections (inherits)
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.
We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
61 views
Does a Registrar forwarding your Domain to your hosting site url, violate Google AdSense TOS?
NameCheap offers to forward traffic to a url of your choosing, which is really handy if you are using a free web hosting service, and/or ever want to move to try another free host someday.
However, I ...
1
vote
1answer
656 views
MSNBot/BingBot not reporting it is a bot?
I have apache logs from my server, and I filter out visits from bots/crawlers/scrapers using a python script that checks for user agent strings containing text like 'bot', 'googlebot', etc.
Lately, ...
3
votes
5answers
370 views
How to make sure that Google indexes your site in less than a day?
I am thinking of creating a website where the content will be only useful for a max of one week. I am also assuming that I will be getting most of the traffic thru search engines basically Google. For ...
3
votes
3answers
217 views
Is my robots.txt working as it should?
I want crawlers to have access to http://www.example.com but not http://www.example.com/
My robots.txt is as follows:
User-agent: *
Allow: /$
Disallow: /
My site is in Google search results, but I ...
1
vote
2answers
98 views
SEO - Crawl Errors
I've been optimizing my website for search engines, and for some reason I have a crawl error on a page that has not been found at 'www.peach-designs.com/a' which is not a page, is not a link, or is ...
7
votes
1answer
427 views
Is there a search engine that indexes source code of a web-page?
I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of ...
2
votes
2answers
1k views
Robots.txt and Regular Expressions
If I place a regular expression in robots.txt will it be readable for the web crawlers?
9
votes
1answer
509 views
How did Google manage to crawl my 403 pages?
I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
6
votes
1answer
228 views
List of all URI elements for a website
Does anyone know a good way of getting a list of all the URI elements for a website? I plan on moving a website to a new CMS and would like to setup some 301 redirects for articles, images, css, and ...
