Tag Info

Hot answers tagged

15

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


14

The most important thing for your 404 pages is the header's status code: it must be 404 not found. It's maybe stupid to say that, but in dynamics applications, with an URI like this http://my.webapp.invalid/index.php?id=4, when id=4 return nothing, many web developers returns a 404 message with a 200 OK status code... An other thing: for example, if ...


14

Strong vote for a dedicated 404 page. Less confusing for the user Allows you to see more easily if something is wrong (through web server error log, your own logging script and/or Google Analytics) You can tailor the page towards the situation (for example display a list of product categories) Allows search engines to clean up their indexes. No longer ...


11

It is impossible to configure logging of the #fragment portion of the URL because your web server never sees it. From RFC3986 the fragment identifier is separated from the rest of the URI prior to a dereference, and thus the identifying information within the fragment itself is dereferenced solely by the user agent, regardless of the URI scheme. ...


11

The correct code would be 401 Not Authorized As per the HTTP specifications 10.4.2 401 Unauthorized The request requires user authentication. The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource. The client MAY repeat the request with a suitable Authorization header ...


10

I often see another site that links to tons of pages on my site that don't exist. Even if you are clicking on that page and not seeing the link: The site might previously have had those links The site may be cloaking and serving those links only to Googlebot and not to visitors It is a waste of resources, but it won't confuse Google and it won't hurt ...


10

Blocking No User Agent Blocking based on no user agent is a silly idea... a lot of users who like to remain anonymous through VPNS will often disable user agent and anything else that can be used to harvest data... And anonymity is growing. Also if the idea behind this is to save on resources it should be noted that most bots that are not legit search ...


9

I also was finding these references in my logs. This seems to be related to a service called "Drop Down Deals", that involves a browser add-on that seems, well, broken. http://www.dropdowndeals.com/ https://nodpi.org/forum/index.php?topic=3462.0;wap2


8

Correct, the crossdomain.xml file is requested to determine if Flash and Silverlight apps are "allowed" to access your website. Personally, I think it's a really dumb convention, but.. it's out there. For Microsoft Silverlight When calling a cross-domain service, Silverlight will check for the existence of clientaccesspolicy.xml first. This is the ...


8

What you're seeing is normal behavior. What you're doing is appending a query string to the home page. If the home page exists, and it always does, it will receive the parameters of the querystring. Now whether or not it uses them or not is up to the page itself. Most pages not expecting a query string will just completely ignore it and display the page ...


8

There is more than Google in this world. A 410 unambiguously tells a bot that the file is gone. A 404 does not. A persistent bot might keep trying to find a 404 indefinitely whereas they might stop trying to find a 410 immediately which would make your server very happy.


7

A custom 404 page is your opportunity to turn a bad experience into a good one. No matter how well built your website is you can't stop someone from typing a URL in their address incorrectly, so you need to be prepared for those users who somehow can't find the content they are looking for. A custom 404 page gives you an opportunity to help that user. A ...


7

Okay. First things first. Do not mark your 404 as being fixed. You are actually prolonging the issue. Google will try and fetch a page that returns a 404 several times before giving up. This is because the 404 error indicates a temporary situation where a 410 error says the page is gone. So every time you mark a 404 as being fixed, you are in effect telling ...


6

To my mind, it will make more sense to use 410 gone status code in this case. The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not ...


6

Echoing the failed URI may enable an XSS attack (it depends on your site) as the URL could be constructed so that a piece of JavaScript is injected and runs on your site. As such this injected JavaScript would have access to the users cookies and if he was logged in it could be used to gain his private information in your system. As long as the URI is ...


6

Google just wrote a blog post that should answer your question very well Q: Do the 404 errors reported in Webmaster Tools affect my site’s ranking? A: 404s are a perfectly normal part of the web; the Internet is always changing, new content is born, old content dies, and when it dies it (ideally) returns a 404 HTTP response code. ...


5

If a piece of content has been intentionally removed from the site, then naturally that page's PR will disappear. That's the appropriate behavior. PR represents votes from other users indicating that they believe that the linked page is valuable. So if the content they voted for has been removed, then naturally their votes should be discarded. Now, you ...


5

It's new crimeware related to exploiting facebook. eg, <script> function fbs_click() { u=location.href; t=document.title; window.open('http://www.facebook.com/sharer.php?u='+encodeURIComponent(u)+'&t='+encodeURIComponent(t),'sharer','toolbar=0,status=0,width=626,height=436'); return false; } </script> <a ...


5

Users first!!! Catering to search engines instead of users is a recipe for failure. Good usability would be to redirect them to a page that explains that the product doesn't exist anymore and invite them to explore other related items. That way you don't lose the visitor due to a bad link or confusion. After all if they are following a link expecting one ...


5

If the page no longer exists, or the URL is invalid, then returning a 404 is fine. In those cases, it's good to see the 404 listed as a crawl error in Webmaster Tools, since that's what it should be showing :-). We did a blog post about this in May with more information: Do 404s hurt my site?


5

I think the only way to do so is using ErrorDocument directive: ErrorDocument 404 index.php because with RewriteRule you are "masking" the error serving another page to the request. Yet you must know that ErrorDocument 404 won't change the URL of the page, thus you will have your homepage on a wrong URL.


5

Besides there being more search engines than Google out there, there's also no reason to assume that Google won't ever change the way they treat 410 responses. Indeed, it seems that's already happened: the information Matt Cutts quotes in the video is from 2007, whereas this post from 2009 by John Mu on Google's Webmaster Central forums says otherwise: ...


5

I've finished what I can on the script so far (read comments of original question for detail and context). Source: http://www.ionfish.org/projects/xml-spider/ Features: ability to start from any point (resume crashed attempt?) since it tells you which number it's processing. scans any sitemap public accessible finds all the links it has to scan BEFORE ...


5

Once you publish a page, Google will never forget about it. I have sites from which I removed pages 15 years ago. Googlebot still comes back and checks those pages occasionally. To prevent the pages from showing up in the search engine, your 404 errors will do the job. It may take Google a day to remove the page from the index after Googlebot crawls it ...


4

Don't worry about them. There are many spammy sites that create links such as that. GoogleBot honours them and tries to follow the URI only to hit a 404. If you do a search for link:http://article-stack.com/2010/08/page/luminous-solutions.net in respective search engines you'll often find the culprit (otherwise they'll have been taken down already). A ...


4

You should use a permanent redirect to tell GoogleBot and others where the page has moved to. If the page no longer exists at all then you can add entries in robots.txt to tell bots not to access the missing page: User-agent: * Disallow: /extraNeus.php


4

put this in .htaccess file (change the path/to/files to your file locations) Redirect 301 /extraNeus.php http://www.yoursite.com/new_page.php you can leave it in there forever or next time google indexes the page you should be good to remove it.


4

You can do what you like with you 404 pages, as Google only retrieves the HTTP header, not the content. No PageRank is assigned to the 404 page, or any of the links contained on/in it. Redirecting to a 404 is merely adding an additional step in the chain (request -> 301/302 -> 404), so there's no value to be gained. Putting links on your 404 pages, ...


4

This isn't actually related to a firefox plugin, or any plugin at all. crossdomain.xml is part of the flash/flex specification. It's a method to allow/validate cross domain operations for flash and other Adobe products, Sliverlight also seems to use/obey the same adobe policy framework. From Adobe's Cross-domain policy file specification A cross-domain ...


4

If your site is truly dynamic then you should not be generating img elements for non-existent images in the first place, thus avoiding 404's from the beginning. Broken img elements are potentially a bad experience for users and for this reason you could be penalised by the search engines. Will search engines see your JavaScript hidden images as "not for ...



Only top voted, non community-wiki answers of a minimum length are eligible