Tag Info

Hot answers tagged

13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


3

You mentioned that two options are valid for you, so I'm using the second one, http://www.testdomain.com/john/2/doe, and assuming a clean .htaccess RewriteEngine On # RewriteCond %{REQUEST_FILENAME} !-f # RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{QUERY_STRING} ^name=([\w]+)\&id=([\d]{1,3})\&nick=([\w]+)$ RewriteRule .* /%1/%2/%3? [R=301,L] ...


3

It is likely that they have edited the .htaccess as root user and your FTP user will not have permissions to override root (no other user will) so unless you also have root access, you'll have to ask them to change the owner of the file to your FTP user so you can edit it.


2

RewriteRule ^.*$ index.php?page=$1 [NC,L] This will result in a rewrite loop. You are also not capturing the sub pattern, so $1 will be empty. In order to prevent a rewrite loop, you need a get-out-clause, such as not rewriting when the request is already for index.php. Something like: RewriteEngine On RewriteCond %{REQUEST_URI} !=/index.php RewriteRule ...


2

You could set up theirsite.com as a parked domain on top of myscript.com. This initially allows theirsite.com to be an alias for myscript.com. You can then use mod_rewrite (in .htaccess) to internally rewrite to the real URL (similar to what you have done already). You can only rewrite to a URL on theirsite.com (not myscript.com), which shouldn't be a ...


2

PHP doesn't work with the file over HTTP but directly on the filesystem, unless you access the file over HTTP using cUrl or file_get_contents('http://.../file.xml'). If you want to prevent the files from being accessed without the user being authenticated first, place the files outside the public directory and serve them from there. /files/ ...


2

I am going to assume you mean that any request for example.com/page.html outputs the content of the file from websitedirectory/page.html. If it was "redirected" then the browser would show the subfolder in the URL. In your case, there is no effect on SEO. Search engines will see the same URLs you see in your browser and won't know about the subdirectory ...


1

Following on from comments... since you are wanting to completely remove these pages from Google's index then simply redirecting (301) them (as requested in your original question) is not necessarily the correct thing to do. Redirection is saying that the page has moved. Yes, Google is likely to drop the original page from the index... eventually, but that ...


1

Maybe this link (how to redirect domain according to country IP address) can help you. I think, however, that this is not a good practice. Redirection by language or country is quite strict. It is better preferred: A message at the top of your site that influence the visitor to go to the page built with his language. A system that can easily change the ...


1

I've had a quick test and this should do the trick: RewriteEngine on RewriteRule ^([0-9]+)/([0-9]+)$ /date/$1/$2 [R=301,L] You may also need to exclude this from the redirect to the index.php page by adding RewriteCond %{REQUEST_URI} !^/(date) [NC] so it should look like this # BEGIN WordPress <IfModule mod_rewrite.c> RewriteEngine On ...


1

The 403 Forbidden error is due to the access denied issue as intended, there is no issue here - that seems fine. The 404 Page Not Found error is due to your .htaccess file looking for and not finding where it expects, some kind of customised error page to show in this circumstance. It's possible your hosting provider may have this configured at the ...


1

Every time I read questions like this I think of Kevin Spacey's character in Henry and June. The fellow who was always writing his greatest Novel, but was so worried about someone stealing his ideas he kept it locked away in a briefcase, carried close to his chest... Every linux user is a "legit" wget user. I use it often for grabbing debs, videos, ...


1

wget is often used for scraping. It's a command-line tool to download webpages and their assets. If your website isn't being publicized, you can almost be sure that it's a bot doing scraping. So yes, you could block it, but also be aware you may need to do something more sophisticated than blocking it with robots.txt since wget can easily be told to ignore ...


1

The best .htaccess code which I am able to give you to do the redirection is as following: # This allows you to redirect your entire website Redirect 301 / http://example.com/ You will have to modify the example as you add it into your .htaccess file.


1

The way I would do this is block all requests which are coming from unwantedcomain.com by checking for HTTP_REFERER Block traffic from a single domain: RewriteEngine on RewriteCond %{HTTP_REFERER} unwateddomain\.com [NC] RewriteRule .* - [F] Block traffic from multiple domains: RewriteEngine on RewriteCond %{HTTP_REFERER} unwanteddomain\.com [NC,OR] ...



Only top voted, non community-wiki answers of a minimum length are eligible