Tag Info

Hot answers tagged

15

As far as I know, no bots or apps request sitemap.xml without being told it should be there. Most sites probably don't have it, and of the sites that do, many use gzip, and many call the file something else or put the sitemaps in a subfolder. Here are all the ones I know of: favicon.ico Gives your pages an icon in tabs, bookmarks, etc. robots.txt Useful ...


14

Strong vote for a dedicated 404 page. Less confusing for the user Allows you to see more easily if something is wrong (through web server error log, your own logging script and/or Google Analytics) You can tailor the page towards the situation (for example display a list of product categories) Allows search engines to clean up their indexes. No longer ...


13

The most important thing for your 404 pages is the header's status code: it must be 404 not found. It's maybe stupid to say that, but in dynamics applications, with an URI like this http://my.webapp.invalid/index.php?id=4, when id=4 return nothing, many web developers returns a 404 message with a 200 OK status code... An other thing: for example, if ...


11

It is impossible to configure logging of the #fragment portion of the URL because your web server never sees it. From RFC3986 the fragment identifier is separated from the rest of the URI prior to a dereference, and thus the identifying information within the fragment itself is dereferenced solely by the user agent, regardless of the URI scheme. ...


9

I also was finding these references in my logs. This seems to be related to a service called "Drop Down Deals", that involves a browser add-on that seems, well, broken. http://www.dropdowndeals.com/ https://nodpi.org/forum/index.php?topic=3462.0;wap2


8

What you're seeing is normal behavior. What you're doing is appending a query string to the home page. If the home page exists, and it always does, it will receive the parameters of the querystring. Now whether or not it uses them or not is up to the page itself. Most pages not expecting a query string will just completely ignore it and display the page ...


7

Correct, the crossdomain.xml file is requested to determine if Flash and Silverlight apps are "allowed" to access your website. Personally, I think it's a really dumb convention, but.. it's out there. For Microsoft Silverlight When calling a cross-domain service, Silverlight will check for the existence of clientaccesspolicy.xml first. This is the ...


7

A custom 404 page is your opportunity to turn a bad experience into a good one. No matter how well built your website is you can't stop someone from typing a URL in their address incorrectly, so you need to be prepared for those users who somehow can't find the content they are looking for. A custom 404 page gives you an opportunity to help that user. A ...


6

To my mind, it will make more sense to use 410 gone status code in this case. The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not ...


6

Echoing the failed URI may enable an XSS attack (it depends on your site) as the URL could be constructed so that a piece of JavaScript is injected and runs on your site. As such this injected JavaScript would have access to the users cookies and if he was logged in it could be used to gain his private information in your system. As long as the URI is ...


6

Google just wrote a blog post that should answer your question very well Q: Do the 404 errors reported in Webmaster Tools affect my site’s ranking? A: 404s are a perfectly normal part of the web; the Internet is always changing, new content is born, old content dies, and when it dies it (ideally) returns a 404 HTTP response code. ...


5

If a piece of content has been intentionally removed from the site, then naturally that page's PR will disappear. That's the appropriate behavior. PR represents votes from other users indicating that they believe that the linked page is valuable. So if the content they voted for has been removed, then naturally their votes should be discarded. Now, you ...


5

It's new crimeware related to exploiting facebook. eg, <script> function fbs_click() { u=location.href; t=document.title; window.open('http://www.facebook.com/sharer.php?u='+encodeURIComponent(u)+'&t='+encodeURIComponent(t),'sharer','toolbar=0,status=0,width=626,height=436'); return false; } </script> <a ...


5

Users first!!! Catering to search engines instead of users is a recipe for failure. Good usability would be to redirect them to a page that explains that the product doesn't exist anymore and invite them to explore other related items. That way you don't lose the visitor due to a bad link or confusion. After all if they are following a link expecting one ...


5

If the page no longer exists, or the URL is invalid, then returning a 404 is fine. In those cases, it's good to see the 404 listed as a crawl error in Webmaster Tools, since that's what it should be showing :-). We did a blog post about this in May with more information: Do 404s hurt my site?


5

I think the only way to do so is using ErrorDocument directive: ErrorDocument 404 index.php because with RewriteRule you are "masking" the error serving another page to the request. Yet you must know that ErrorDocument 404 won't change the URL of the page, thus you will have your homepage on a wrong URL.


5

Besides there being more search engines than Google out there, there's also no reason to assume that Google won't ever change the way they treat 410 responses. Indeed, it seems that's already happened: the information Matt Cutts quotes in the video is from 2007, whereas this post from 2009 by John Mu on Google's Webmaster Central forums says otherwise: ...


5

I've finished what I can on the script so far (read comments of original question for detail and context). Source: http://www.ionfish.org/projects/xml-spider/ Features: ability to start from any point (resume crashed attempt?) since it tells you which number it's processing. scans any sitemap public accessible finds all the links it has to scan BEFORE ...


4

Don't worry about them. There are many spammy sites that create links such as that. GoogleBot honours them and tries to follow the URI only to hit a 404. If you do a search for link:http://article-stack.com/2010/08/page/luminous-solutions.net in respective search engines you'll often find the culprit (otherwise they'll have been taken down already). A ...


4

You can do what you like with you 404 pages, as Google only retrieves the HTTP header, not the content. No PageRank is assigned to the 404 page, or any of the links contained on/in it. Redirecting to a 404 is merely adding an additional step in the chain (request -> 301/302 -> 404), so there's no value to be gained. Putting links on your 404 pages, ...


4

This isn't actually related to a firefox plugin, or any plugin at all. crossdomain.xml is part of the flash/flex specification. It's a method to allow/validate cross domain operations for flash and other Adobe products, Sliverlight also seems to use/obey the same adobe policy framework. From Adobe's Cross-domain policy file specification A cross-domain ...


4

If your site is truly dynamic then you should not be generating img elements for non-existent images in the first place, thus avoiding 404's from the beginning. Broken img elements are potentially a bad experience for users and for this reason you could be penalised by the search engines. Will search engines see your JavaScript hidden images as "not for ...


4

Let's analyse your examples. All three have on thing in common: /enquiry/frmenquiry.aspx. That makes it worth googling for. One of the results is this question, but there are several others, a few of which actually display the HTML source code instead of the rendered HTML. That is a classic attack vector for compromising web sites. I would surmise that your ...


4

You should login to Google Webmaster Tools and do a fetch as Google, if the page returns a status 200 then you know that your pages are working as intended and what your experiencing is just the Google cache service not working, which should resolve itself in time. Google Cache is not Realtime It's worth mentioning that the Google cache system runs ...


4

301 Redirecting is generally a bad idea when the pages are not relevant. It is believed that Google now determine relevant keywords on the page of links to that of the linked page. Too many non-relevant links you run the risk of being slapped by Google. Also it's considered bad user experience for users to be greeted with a page they didn't expect ...


4

There are a few possible reasons why a bot would try to visit a removed page: The bot followed a link to that page from another website. Bots frequently omit referrer so it is difficult to tell if this is the case. Given that the bot in question has "backlink checker" as part of its tagline, this seems a likely cause. The bot had visited the page while it ...


3

How bad is it? Badness is directly proportional to the number of links to the 404 and the number of visitors who are disappointed when they arrive at your site; Google Webmaster Tools won't show you a 404 unless there's a link to the content somewhere on the internet. Does anyone know how the spider acts programatically with crawl errors[?] If ...


3

I found the answer with the SM36_3M.jpg file, you need to make sure the extension is uppercase: http://ercolehome.com/images/products/mirrors/SM36_3M.JPG The way I found this out was by going to here http://ercolehome.com/images/products/mirrors/ You might want to turn off directory listing.


3

As others have said, page anchors are something that the browser operates on rather than the web server. You could use some code to pass the anchor info on to your Analytics program. For example you could extract the location.hash value and pass it to Google Analytics as a custom variable. More usefully you could run code that checks the page html for ...


3

I'd vote for a separate error page if only to let the users of your site know that the link they clicked failed or they typed the wrong URL. It would be confusing to get back to the home page. I know I'd be tempted to repeat my actions to see if I'd done something wrong! If you added a "report this broken link" link or even logged it yourself, you'd get ...



Only top voted, non community-wiki answers of a minimum length are eligible