Tag Info

Hot answers tagged

13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


7

You need to drop the 'http:' prefix from all image (and any other) links on your site. When you do so, your browser will default to whatever protocol the page loads with. For example this is best: <a href="//www.example.com/"> And this is not: <a href="http://www.example.com/"> The top example is perfectly valid markup and you will load all ...


7

Empty Virtual Host With virtual hosting, all traffic is routed to an IP address and then Apache matches the hostname. When virtual hosting using NameVirtualHost is enabled, the site that responds to the IP address is the first one listed in the Apache configuration file. So you can use a null virtualhost: <VirtualHost 192.168.1.1:80> ...


6

The way that you will do the 301 redirect does not play any role for the Google. Actually it does not see your configuration file. What it is important is to send the 301 headers with the new location on each old url and not just the home page. For example if you have 10 pages on the old website, make sure that these all pages are redirected to the new 10 ...


5

Just redirect by 301 HTTP status all your old URLs to the new ones. If you only change file extension, you can do it easily with an .htaccess file (if you use Apache as a web server). Put these lines in your .htaccess file: RedirectMatch 301 (.*)\.html$ http://www.example.com$1.php Of course, change www.example.com by your domain name. By using 301 ...


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


5

Although the browser should cache permanent redirects, I think an internal rewrite would be preferable. This avoids the browser having to make an additional HTTP request: RewriteEngine On RewriteRule ^([a-z]+\.css)$ /css/$1 [L]


4

If you can, it's a better idea to put the redirects from the old urls to their new equivalent. For any other pages that have no new equivalent or are being discarded, you can either leave those to be 404, or redirect those to the home page. It's also a good idea to have a custom 404 page for the new version of the site that contains a simplified site map, ...


4

You're almost there. Drop the very first RewriteCond (as it's not required and won't match ever) and add a trailing / to your last RewriteRule since you're redirecting with them. RewriteEngine On RewriteCond %{HTTP:Accept-Language} ^en [NC] RewriteRule ^$ http://mysite.com/en/ [L,R=301] RewriteCond %{HTTP:Accept-Language} ^de [NC] RewriteRule ^$ ...


4

The "hashbang" or "shebang" url syntax is an old hack designed to provide browser history to AJAX sites and to help Google index content that is loaded via JavaScript. See "What's the shebang/hashbang (#!) in Facebook and new Twitter URLs for?" for more. Your WordPress theme, "John Smith", is responsible for rewriting URLs in this way. It does this so that ...


4

To prevent the image being indexed via the site you are hotlinking from, instead of linking (hotlinking) directly to the source image you could perhaps call a script which reads and serves the appropriate image from the source site together with an X-Robots-Tag: noindex HTTP response header. <img src="/get-image.php?file=my-image-hosted-elsewhere.jpg" ...


4

Thanks! On further investigation, I also found the hint to add <meta name="robots" content="noimageindex"> to the header of the web page - it is supposed to prevent the page used as the referring page according to a few sources. I'll see if that's sufficient and update my post if necessary.


4

Bingbot is not "crashing". The "Fetch as Bingbot" tool within Bing's Webmaster Tools simply does not follow redirects: From bing webmaster help: WHAT DOES "REDIRECTION LIMIT REACHED MEAN"? Unlike the SEO Analyzer tool, Fetch as Bingbot does not follow redirects. Instead it will let you know that the page resulted in a redirect and shows you the HTTP ...


4

Put these lines in your .htaccess and PageSpeed Insights will see your gzip compression: <IfModule mod_deflate.c> <FilesMatch "\.(html|php|txt|xml|js|css)$"> SetOutputFilter DEFLATE </FilesMatch> </IfModule> It works for my sites.


4

You could check the $_SERVER['REQUEST_URI'] on the php script restaurants.php if the request url is the correct. For example: if ($_SERVER['REQUEST_URI'] !== '/' . $name . '/' . $id) { header ('HTTP/1.1 301 Moved Permanently'); header ('Location: http://www.bgmenus.com/' . $name . '/' . $id); die(); } Where $name is the restaurant name ...


4

You could do this using the following rewrite rule in .htaccess: RewriteEngine on RewriteCond %{QUERY_STRING} searchkey RewriteRule .* - [G,L] .* means that it will do this for any URL on your site (as long as the condition of having "searchkey" in the query string is met) - means to leave the URL alone and not change it G means "Gone" -- send the 410 ...


4

You can use one of these solutions: Keep the Current .html extension unchanged for just frontend purpose and use Apache .htaccess file to proxify your requests so that. Add this to your .htaccess file to work: RewriteEngine on RewriteRule ^(.*).html $1.php OR Why not just have the HTML pages parsed as PHP? That way you can get the best of ...


4

To block all "blank" User-Agents or User-Agents consisting of a hyphen, you could use the following in your .htaccess RewriteCond %{HTTP_USER_AGENT} ^-?$ RewriteRule ^ - [F]


4

You could just redirect /styles.css to /css/styles.css. In the .htaccess in the web root: RedirectMatch permanent /([a-z]+\.css)$ http://example.com/css/$1 (Replacing example.com with your real domain name.)


4

Activate mod_rewrite, and put this in a .htaccess file or a <Directory> directive: RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-l RewriteCond %{REQUEST_URI} !^apps/ RewriteRule (.*) apps/$1 What it does: The first three lines check if the requested URI refers to an actually existing file, ...


4

Regarding: "Is this ok?" No, its not good/intuitive for your users and hence not good for SEO. You should be using something like this www.alanmarth.com/ (Main Page) www.alanmarth.com/servicios (Services) www.alanmarth.com/blog (Recent news) www.alanmarth.com/blog/nameOfCategory2 (News category) www.alanmarth.com/blog/titleOfBlog3 (A single entry) ...


3

I met the same problem today. I solved it in two steps Enabling of rewrite module sudo a2enmod rewrite I changed AllowOverride None to AllowOverride All in my virtual host configuration file. AllowOverride None - prevents Apache server from reading of any .htaccess file located in your site directories. Hope it helps.


3

Well the correct way in your case (which is: the user consciously requests an invalid resource) is to return a 404-response code. The body of the response is a description for the user of your site. If your 404.php is just like a template and contains mostly HTML-code then you could just include it. Otherwise showing the "project '$name' does not exist" is ...


3

You need to enable the rewrite engine in your .htaccess file if you have not done already... RewriteEngine on RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC] RewriteRule ^(.*)$ http://example.com/$1 [L,R=301] Note that I've removed the extra .com on the RewriteCond line that you have in your question - I'm assuming this was a typo? Also, FollowSymLinks ...


3

You've made the redirect correctly. It works well. However you must wait until Google recrawl and reindex it before it appears on search results.


3

All of the following code will be placed in a file called .htaccess in your root web directory and mod_rewrite enabled on your server. Changing the domain is easy to do. The following snippet should do it: Options +FollowSymlinks RewriteEngine On RewriteCond %{HTTP_HOST} ^(www\.)?old\.com$ [NC] RewriteRule ^(.*)$ http://www.new.com/$1 [R=301,L] To move ...


3

the $ anchor signals the end of the subject (as ^ signals the start), thus your expr is incorrect. What you're lookng for is more likely something like # to rewrite images/foo to image.php?id=foo RewriteRule ^images/(.*)$ image.php?id=$1 [QSA,L] # to rewrite foo/images to image.php?id=foo RewriteRule ^(.*)/images$ image.php?id=$1 [QSA,L] For more info ...


3

The pseudo-code translation of your .htaccess file would be something along these lines: Line 1: In case we weren't previously planning to do anything special with URLs, we are now (RewriteEngine is an optional processing module and we're making sure its enabled). Line 2: When we're talking about rewrite URLs, from here on append the path /, which in your ...


3

Finally resolved using these rewrite rules: RewriteEngine on RewriteCond %{HTTP_HOST} ^www.unwanteddomain.com$ [NC] RewriteRule .* http://whateverPlaceYouWantToSend.com [R,L] HTTP_REFERER did not work so I used HTTP_HOST.


3

It is likely that they have edited the .htaccess as root user and your FTP user will not have permissions to override root (no other user will) so unless you also have root access, you'll have to ask them to change the owner of the file to your FTP user so you can edit it.



Only top voted, non community-wiki answers of a minimum length are eligible