Tag Info

Hot answers tagged

13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


7

You need to drop the 'http:' prefix from all image (and any other) links on your site. When you do so, your browser will default to whatever protocol the page loads with. For example this is best: <a href="//www.example.com/"> And this is not: <a href="http://www.example.com/"> The top example is perfectly valid markup and you will load all ...


7

Empty Virtual Host With virtual hosting, all traffic is routed to an IP address and then Apache matches the hostname. When virtual hosting using NameVirtualHost is enabled, the site that responds to the IP address is the first one listed in the Apache configuration file. So you can use a null virtualhost: <VirtualHost 192.168.1.1:80> ...


6

The way that you will do the 301 redirect does not play any role for the Google. Actually it does not see your configuration file. What it is important is to send the 301 headers with the new location on each old url and not just the home page. For example if you have 10 pages on the old website, make sure that these all pages are redirected to the new 10 ...


5

Just redirect by 301 HTTP status all your old URLs to the new ones. If you only change file extension, you can do it easily with an .htaccess file (if you use Apache as a web server). Put these lines in your .htaccess file: RedirectMatch 301 (.*)\.html$ http://www.example.com$1.php Of course, change www.example.com by your domain name. By using 301 ...


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


5

Although the browser should cache permanent redirects, I think an internal rewrite would be preferable. This avoids the browser having to make an additional HTTP request: RewriteEngine On RewriteRule ^([a-z]+\.css)$ /css/$1 [L]


5

No this is not good. Google did not intend for nofollow to be used for internal links. It is meant to be used for links that you do not have editorial control over. This is going to hurt you a lot. Your .htaccess code is not adding nofollow to your links. It is not possible for .htaccess to alter your HTML like that. Something else is causing your problem. ...


4

Bingbot is not "crashing". The "Fetch as Bingbot" tool within Bing's Webmaster Tools simply does not follow redirects: From bing webmaster help: WHAT DOES "REDIRECTION LIMIT REACHED MEAN"? Unlike the SEO Analyzer tool, Fetch as Bingbot does not follow redirects. Instead it will let you know that the page resulted in a redirect and shows you the HTTP ...


4

Thanks! On further investigation, I also found the hint to add <meta name="robots" content="noimageindex"> to the header of the web page - it is supposed to prevent the page used as the referring page according to a few sources. I'll see if that's sufficient and update my post if necessary.


4

To prevent the image being indexed via the site you are hotlinking from, instead of linking (hotlinking) directly to the source image you could perhaps call a script which reads and serves the appropriate image from the source site together with an X-Robots-Tag: noindex HTTP response header. <img src="/get-image.php?file=my-image-hosted-elsewhere.jpg" ...


4

Put these lines in your .htaccess and PageSpeed Insights will see your gzip compression: <IfModule mod_deflate.c> <FilesMatch "\.(html|php|txt|xml|js|css)$"> SetOutputFilter DEFLATE </FilesMatch> </IfModule> It works for my sites.


4

You could check the $_SERVER['REQUEST_URI'] on the php script restaurants.php if the request url is the correct. For example: if ($_SERVER['REQUEST_URI'] !== '/' . $name . '/' . $id) { header ('HTTP/1.1 301 Moved Permanently'); header ('Location: http://www.bgmenus.com/' . $name . '/' . $id); die(); } Where $name is the restaurant name ...


4

You could do this using the following rewrite rule in .htaccess: RewriteEngine on RewriteCond %{QUERY_STRING} searchkey RewriteRule .* - [G,L] .* means that it will do this for any URL on your site (as long as the condition of having "searchkey" in the query string is met) - means to leave the URL alone and not change it G means "Gone" -- send the 410 ...


4

You can use one of these solutions: Keep the Current .html extension unchanged for just frontend purpose and use Apache .htaccess file to proxify your requests so that. Add this to your .htaccess file to work: RewriteEngine on RewriteRule ^(.*).html $1.php OR Why not just have the HTML pages parsed as PHP? That way you can get the best of ...


4

To block all "blank" User-Agents or User-Agents consisting of a hyphen, you could use the following in your .htaccess RewriteCond %{HTTP_USER_AGENT} ^-?$ RewriteRule ^ - [F]


4

You could just redirect /styles.css to /css/styles.css. In the .htaccess in the web root: RedirectMatch permanent /([a-z]+\.css)$ http://example.com/css/$1 (Replacing example.com with your real domain name.)


4

Activate mod_rewrite, and put this in a .htaccess file or a <Directory> directive: RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-l RewriteCond %{REQUEST_URI} !^apps/ RewriteRule (.*) apps/$1 What it does: The first three lines check if the requested URI refers to an actually existing file, ...


4

Regarding: "Is this ok?" No, its not good/intuitive for your users and hence not good for SEO. You should be using something like this www.alanmarth.com/ (Main Page) www.alanmarth.com/servicios (Services) www.alanmarth.com/blog (Recent news) www.alanmarth.com/blog/nameOfCategory2 (News category) www.alanmarth.com/blog/titleOfBlog3 (A single entry) ...


3

All of the following code will be placed in a file called .htaccess in your root web directory and mod_rewrite enabled on your server. Changing the domain is easy to do. The following snippet should do it: Options +FollowSymlinks RewriteEngine On RewriteCond %{HTTP_HOST} ^(www\.)?old\.com$ [NC] RewriteRule ^(.*)$ http://www.new.com/$1 [R=301,L] To move ...


3

Currently my website is under maintenance. If your website is only temporarily "under maintenance" and has already been live and indexed by search engines then you should consider returning a "503 Service Unavailable" HTTP response code with perhaps a Retry-After HTTP header indicating when the site is expected to be available again. Instead of simply ...


3

the $ anchor signals the end of the subject (as ^ signals the start), thus your expr is incorrect. What you're lookng for is more likely something like # to rewrite images/foo to image.php?id=foo RewriteRule ^images/(.*)$ image.php?id=$1 [QSA,L] # to rewrite foo/images to image.php?id=foo RewriteRule ^(.*)/images$ image.php?id=$1 [QSA,L] For more info ...


3

The pseudo-code translation of your .htaccess file would be something along these lines: Line 1: In case we weren't previously planning to do anything special with URLs, we are now (RewriteEngine is an optional processing module and we're making sure its enabled). Line 2: When we're talking about rewrite URLs, from here on append the path /, which in your ...


3

Finally resolved using these rewrite rules: RewriteEngine on RewriteCond %{HTTP_HOST} ^www.unwanteddomain.com$ [NC] RewriteRule .* http://whateverPlaceYouWantToSend.com [R,L] HTTP_REFERER did not work so I used HTTP_HOST.


3

It is likely that they have edited the .htaccess as root user and your FTP user will not have permissions to override root (no other user will) so unless you also have root access, you'll have to ask them to change the owner of the file to your FTP user so you can edit it.


3

I do not believe so. Not using ErrorDocument the way you have specified. Here is a resource site that seems to have some answers. http://www.askapache.com/htaccess/htaccess.html#Custom_ErrorDocuments Scroll down and you will find this example. If it is not exactly what you want, it can be modified. ### ALTERNATATIVE TO USING ERRORDOCUMENT # ...


3

You do not need a custom error document in order to monitor 404's on your site - if that is the requirement. All the information is in your server access log. The HTTP response code (ie. 404, 403, 200, etc) and the URL of the request that produced that response. The custom error document is a nice way of serving a meaningful response to users. You can also ...


3

Check here: http://stackoverflow.com/questions/9249243/how-to-disable-apache-gzip-compression-for-some-media-files-in-htaccess-file It's possible, but looks like it will disable gzip for the whole website. Keep in mind that you are telling PHP not to gzip, but Apache may still be gzipping so this should sort it, but check Apache config. RewriteRule ^(.*)$ ...


3

I ran a quick test and got the following in my Apache access logs from adding an image to a Google Spreadsheet: 64.233.172.188 - - [09/May/2014:05:14:51 -0700] "GET twitter.png HTTP/1.1" 200 1842 "-" "Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)" 66.249.80.216 - - [09/May/2014:05:14:52 -0700] "GET twitter.png ...


3

Assuming you mean "subdirectory" and not "subdomain" (as per your examples), then try something like the following in the .htaccess file at x.com: RewriteEngine On RewriteCond %{REQUEST_URI} !^/sub/ RewriteRule (.*) http://y.com/$1 [R=301,L] Redirect (301) everything to y.com where the URI does not start with /sub/. Note that this redirects to the ...



Only top voted, non community-wiki answers of a minimum length are eligible