Tag Info

Hot answers tagged

124

Disclaimer: I work together with the Sitemaps team at Google, so I'm somewhat biased :-). In addition to using Sitemaps extensively for "non-web-index" content (images, videos, News, etc.) we use information from URLs included in Sitemaps files for these main purposes: Discovering new and updated content (I guess this is the obvious one, and yes, we do ...


78

A Sitemap file helps search engines to discover new and updated URLs on your website. In particular, if your website is fairly large, then this can help them to be able to focus on the new & updated content, instead of having to blindly crawl through everything to see if anything has changed. That can result in new content being found much faster, which ...


34

If you know you have good site architecture and the Google would find your pages naturally the only benefit I'm aware of is faster indexing, if your site is getting indexed fast enough for you then no need. Here's article from 2009 where a gentlemen tested how fast Google crawled his site with a sitemap and without. ...


14

I suspect: for Google, sitemaps are necessary to keep track of updates in the fastest way possible. E.g., let's say you have added a new content to some deep location of your web site, which takes more than 10-20 clicks to reach from your home page. For Google to reach this new page would be less likely in a short time - so instead, until a path to this page ...


14

In your header you have a canonical link (on line 11, just under <title>). It looks like this on your page: <link href="http://escene.ir/component/products/?task=view.12" rel="canonical" /> This element tells Google your preferred URL for a page which has several urls to choose from. This is to prevent you from being penalized for having ...


13

Do 301 redirects from the old URLs to the new URLs. This will tell the search engines and users that the pages have moved and where they are now. This also associates the old URL with the new URL for Google which means all of your old incoming links will now be associated with your new pages. Submit an XML sitemap in Google Webmaster Tools Make sure you ...


12

Include all pages. The purpose of the XML sitemap is to tell the search engines about all of your content. Not just the new stuff. From the sitemaps.org website (emphasis mine): Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. If you have a lot of content you can use ...


12

No, it doesn't matter, and it's actually a pretty good idea if you have regularly added content. It makes no difference to Google whether you wrote the file by hand, generated it manually with an online tool, generated it automatically with a cronjob, or generated it live each time the sitemap is requested. Edit: As danlefree suggests, you ought to consider ...


11

Sitemap Limits: Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap. Sitemaps can be compressed using gzip, reducing bandwidth consumption. Multiple sitemap files are supported, with a Sitemap index file serving as an entry point for a total of 1000 Sitemaps. Note that you can specify multiple sitemaps in your ...


11

We now utilize a crawling method of adding new content to our search index. The subdomain you refer to was just retired over the weekend, and we'll be updating that FAQ page very shortly. Best regards, Eric from Ask.com


10

See About Sitemaps, from Google: Sitemaps are particularly helpful if: Your site has dynamic content. Your site has pages that aren't easily discovered by Googlebot during the crawl process—for example, pages featuring rich AJAX or images. Your site is new and has few links to it. (Googlebot crawls the web by following links from one ...


10

Multiple sitemaps are perfectly fine. There's a special way to tell the search engines you have them. <?xml version="1.0" encoding="UTF-8"?> <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap> <loc>http://www.example.com/sitemap1.xml.gz</loc> ...


10

It looks like Google was having some technical crawl problems this week, that sound remarkably like what we were experiencing: http://searchengineland.com/is-google-broken-sites-big-small-seeing-indexing-problems-53701 No one seems to be immune from a Google indexing problem that has many site owners baffled. Blogs and websites, big and small, aren’t ...


10

If you want to submit your sitemap without a Google Webmaster Tool account, place it in your robots.txt file. Source You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line including the full URL to the sitemap: Sitemap: http://www.example.com/sitemap.xml This directive is independent of ...


8

For sitemap generation I use GSiteCrawler it works great and it gives you a lot of additional information. If you're looking to do additional analysis I use Xenu link Sleuth it's main purpose is to help you find broken links but it also gives a lot of IA related info such as depth from the home page, internal links pointing at certain pages etc. Both tools ...


8

In Google's words: "In most cases, webmasters will benefit from Sitemap submission, and in no case will you be penalized for it." But I agree that the best thing you can do if you want your website pages to appear in search engines is to make sure they are crawlable from the site proper.


8

Sitemaps are incredibly valuable if you use them correctly. First off, the fact that Google says they are hints is only there to a) ensure that webmasters aren't under the false impression that sitemap = indexation and b) give Google the ability to ignore certain sitemaps if they deem them to be unreliable (aka lastmod is the current date for all URLs each ...


8

It shouldn't have any effect on your pages' rankings. After all, its purpose is to tell the search engines what pages you would like to have indexed and where to find them. It has nothing to do with relevancy. But you see things like the Sitemap Paradox and realize it is far from a perfect system. Having said that, the odds are it isn't your sitemap that ...


8

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature. You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...


7

If you know how to do any server side programming you can write your own dynamic sitemap script which pulls the latest information from your database each time it is called. If your site doesn't update too frequently then you can cache the results and only update when necessary. update Unlimited Sitemap Generator (Not free) This one can handle large sites ...


7

That is correct. It is the HTML entity for an ampersand (&) and is the proper character representation of it in a properly encoded URL. Ampersands (&) and well as < and > are special characters in XML and HTML and need to be displayed using their special character entities.


7

Google doesn't make any offer or guarantee that pages in a sitemap will be indexed. My experience has been that a page has to be linked-to (from a page of some authority) to show up. Is that page/question linked to directly/indirectly from a page with some authority? E.g. if the superuser.com homepage (which presumably has many inlinks) linked directly ...


7

I believe that search engines use the sitemap not so much to find pages, but to optimize how they often they check them for updates. They look at <changefreq> and <lastmod>. Google probably spiders the entire website very often (check your logs!), but not all search engines have the resources to do that (Has anyone tried Blekko?). In any case ...


7

Declaring Language in XHTML and HTML The W3 describes how to formally declare the language of your web pages: http://www.w3.org/International/tutorials/language-decl/ Add a content-language meta tag to the head block of each page. <meta http-equiv="content-language" content="en"> ...or... <meta http-equiv="content-language" content="es"> ...


7

There is no rule that says a sitemap file should live in the root of a website and be named in a special way. Any engine that allows you to submit XML sitemaps allows you to specify the URL for the sitemap. I usually put them in a /sitemaps/ folder for convenience because I usually have several sitemaps. One for videos, one for news articles and one for ...


7

No but I would use the canonical links as much as possible. That way they get copied and passed along right. So if the canonical has the slug, I suggest you use that one. Actually, I high recommend canonical to use the slug, it will be more search-engine friendly.


7

For reasons why you should (or should not) use the www subdomain prefix then see this related post on StackOverflow which raises some good points for both cases. However, it is untrue to say it has been "deprecated" in any form - perhaps somewhat fallen out of favour, but in no way officially deprecated. When it comes to SEO the main thing is to only use ...


7

Yes and No. From the sitemap.org official site:- The location of a Sitemap file determines the set of URLs that can be included in that Sitemap. A Sitemap file located at http://example.com/catalog/sitemap.xml can include any URLs starting with http://example.com/catalog/ but can not include URLs starting with http://example.com/images/. So ...


7

In XML the ampersand has to be escaped, so change it to: &amp;



Only top voted, non community-wiki answers of a minimum length are eligible