All Questions
Tagged with web-crawlers htaccess
25 questions
3
votes
0
answers
731
views
Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server
My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
- 195
1
vote
1
answer
386
views
Blocking bad bots based on useragent in apache conf
I am proposing blocking bad bots using the following code in /etc/sites-enabled/site.conf and I have a few questions.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} useragent1|useragent2|useragent3 [...
- 11
3
votes
2
answers
1k
views
How to prevent hotlinking for specific domains using .htaccess?
I'm being heavily spammed by image scraper sites. There are more than 150+ domains targeting my images & sending spammed links to my site. I just want to prevent hotlinking from them. I just want ...
- 39
2
votes
1
answer
69
views
Do search engines crawl links with query strings?
If I were to have a url like https://example.com/books, that contained several links to urls such as:
https://example.com/books?id=foo
https://example.com/books?id=bar
https://example.com/books?id=...
- 227
4
votes
3
answers
2k
views
Blocking bad bots in .htacess - should I add a BLOCK list or ALLOW list?
I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and ...
- 397
2
votes
1
answer
1k
views
htaccess empty referer deny "google bot"
I put this rule into htaccess file to deny empty referers which is returning 403
SetEnvIfNoCase Referer "^$" bad_user
Deny from env=bad_user
I can see the log, it's denying also Googlebot which ...
1
vote
1
answer
68
views
If .htaccess is used to block my bot from accessing a particular directory, will I know this?
I'm working on a research project and I have a question.
Say I would like to crawl all pages of a given site. In the case that my bot is blocked from accessing a certain portion of the site, I would ...
- 171
1
vote
0
answers
42
views
How does a webmaster perform a proper redirect from one domain to another? [closed]
There are a few questions that are inherently present inside the main question. Those questions are:
What is the best way to perform this type of redirect? (Presumably a 301 Perma for SEO Best ...
- 55
2
votes
1
answer
49
views
Allow crawlers to pass link juice from similar and duplicated user created pages
My users commonly create new pages with similar/duplicate content. For example:
http://example.com/post?123
http://example.com/post?456
I've therefore disallowed these pages in the robots.txt file.
...
- 123
16
votes
4
answers
5k
views
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
- 405
2
votes
2
answers
6k
views
redirecting bots and crawlers to another but not human via htaccess
I would to apply this diagram via htaccess I tried a lots of codes but I failed every time
So I need to redirecting bots and crawlers especially from facebook via .htaccess
- 29
1
vote
1
answer
280
views
How to keep friendly URLs consistent?
I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed.
The original site ran on a combination of Drupal and a lesser known, ...
- 323
1
vote
0
answers
29
views
URL Dynamically changes after first edit
We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages.
Our condition is if any miss-spelled URLs entered directly ...
- 142
0
votes
1
answer
668
views
Is it a good practice to block spam bots by IP ranges?
I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall.
Is ...
- 257
0
votes
0
answers
15
views
spam referers bypass htaccess rules [duplicate]
Here is a sample code I'm using in .htaccess to block spam referers:
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://.*buttons-for-your-website\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://....
- 257
2
votes
3
answers
5k
views
How to block a user-agent that has a spacing in its name?
I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in ...
- 257
5
votes
1
answer
6k
views
Redirect Google crawler to different robots.txt via .htaccess
I have googled for the answer all day and still couldn't find an answer.
I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
- 61
1
vote
1
answer
627
views
How to block a spam bot .htaccess proper formatting?
I have downloaded my access log file from my cPanel and found I have a bot I would like to stop. In the access log, there User-Agent string reads the following;
88.80.205.219 - - [08/May/2014:23:29:...
- 11
5
votes
5
answers
4k
views
Should I block agent Wget/1.12 (linux-gnu)?
Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
- 133
1
vote
1
answer
2k
views
301 redirect causing Bingbot to crash, redirection limit reached
Inside my .htaccess file, I have the following script:
RewriteEngine on
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
<IfModule mod_rewrite.c>
...
- 115
5
votes
2
answers
301
views
Prevent Google from listing hotlinked images for other sites
I am hosting several domains on my server and hotlink images accross various domains.
However, I have found out that in Google searches, the images are listed as originating on the site where the ...
- 285
3
votes
1
answer
2k
views
How to make "crawl-delay" works for all "user-agent" in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
- 193
4
votes
2
answers
699
views
Can the .htaccess file slow down a website to a crawl? If so, are there better ways to solve these problems with different rewrite rules and such?
here is my htaccess file......
RewriteCond %{REQUEST_URI} ^/patients/billing/FAQ_billing\.html$ [OR]
RewriteCond %{REQUEST_URI} ^/patients/billing/getintouch\.html$
RewriteRule ^patients/...
- 53
2
votes
1
answer
938
views
How to hide download file from bots? [duplicate]
Possible Duplicate:
How to restrict the download of all files in a folder?
I want to make a private file available for download but not use username/password protection. I want to put the file ...
- 363
1
vote
1
answer
396
views
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site.
What's the symptom? Google + Bing crawl, but don't index any of the pages.
Let me be clear: I'm ...
- 13