Tag Info

Hot answers tagged

125

Disclaimer: I work together with the Sitemaps team at Google, so I'm somewhat biased :-). In addition to using Sitemaps extensively for "non-web-index" content (images, videos, News, etc.) we use information from URLs included in Sitemaps files for these main purposes: Discovering new and updated content (I guess this is the obvious one, and yes, we do ...


78

A Sitemap file helps search engines to discover new and updated URLs on your website. In particular, if your website is fairly large, then this can help them to be able to focus on the new & updated content, instead of having to blindly crawl through everything to see if anything has changed. That can result in new content being found much faster, which ...


34

If you know you have good site architecture and the Google would find your pages naturally the only benefit I'm aware of is faster indexing, if your site is getting indexed fast enough for you then no need. Here's article from 2009 where a gentlemen tested how fast Google crawled his site with a sitemap and without. ...


14

I suspect: for Google, sitemaps are necessary to keep track of updates in the fastest way possible. E.g., let's say you have added a new content to some deep location of your web site, which takes more than 10-20 clicks to reach from your home page. For Google to reach this new page would be less likely in a short time - so instead, until a path to this page ...


13

Do 301 redirects from the old URLs to the new URLs. This will tell the search engines and users that the pages have moved and where they are now. This also associates the old URL with the new URL for Google which means all of your old incoming links will now be associated with your new pages. Submit an XML sitemap in Google Webmaster Tools Make sure you ...


10

No Robots Exclusion Protocol compliant search engine may crawl any URL disallowed in robots.txt, no matter where else it might be listed. However, Google doesn't necessarily have to crawl your URLs in order to index them. If they believe they have sufficient evidence that there actually is a page at that URL (and a sitemap listing very likely counts as ...


8

Sitemaps are incredibly valuable if you use them correctly. First off, the fact that Google says they are hints is only there to a) ensure that webmasters aren't under the false impression that sitemap = indexation and b) give Google the ability to ignore certain sitemaps if they deem them to be unreliable (aka lastmod is the current date for all URLs each ...


8

In Google's words: "In most cases, webmasters will benefit from Sitemap submission, and in no case will you be penalized for it." But I agree that the best thing you can do if you want your website pages to appear in search engines is to make sure they are crawlable from the site proper.


8

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature. You can specify the name of the sitemap file in robots.txt. It doesn't have to be ...


8

Sitemaps not sitemap when dealing with high volume of content Your first mistake would to assume that you use 'a sitemap', when dealing with sites that contain a lot of content you need to think in terms of 'sitemaps'. Google indexes content without a sitemap Now it's worth mentioning that Google will pretty much index any important URL without the need ...


7

I believe that search engines use the sitemap not so much to find pages, but to optimize how they often they check them for updates. They look at <changefreq> and <lastmod>. Google probably spiders the entire website very often (check your logs!), but not all search engines have the resources to do that (Has anyone tried Blekko?). In any case ...


7

For reasons why you should (or should not) use the www subdomain prefix then see this related post on StackOverflow which raises some good points for both cases. However, it is untrue to say it has been "deprecated" in any form - perhaps somewhat fallen out of favour, but in no way officially deprecated. When it comes to SEO the main thing is to only use ...


6

The Google help page on submitting sitemaps says: "Note: If you are providing a Sitemap index file, you only need to issue a single HTTP request that includes the location of the Sitemap index file; you don't need to issue individual requests for each Sitemap listed in the index." Assuming that you're indeed providing correct lastmod timestamps in your ...


6

If you're not getting any errors then you can assume Google has parsed it and is aware of the contents. But that doesn't mean they will crawl and/or index those pages. Sitemaps are just another way to tell search engines about your pages. They are not obligated to crawl and index any or all of those pages. The same applies to them finding pages through links ...


6

You are asking two questions here. Does a sitemap need to be XML? The simple answer is no, it doesn't have to be XML. It can be XML file, a text file or RSS/Atom feed (which is basically XML), HTML Sitemap HTML Sitemaps: These are used on your website to display the layout in layers on your website to any customer that would wish too (don't know why they ...


6

The problem is that if you (quite rightly) want your content to be indexed by search engines, anyone who performs a site: search in one of the search engines will be able to see what URLs are indexed. If you want to "hide" your sitemap you could have it on a URL with a "secret" name so it's not obvious to anyone who may be looking for it, but seeing as ...


6

you should use both: crawling encouragement using a sitemap.xml – A well structured sitemap.xml can help search engine spiders to find your content quick and directly. And a sitemap.xml gives you the option, of curating what you want to have crawled, how often you consider crawling useful and you can even put a focus on content you find important – the ...


5

if you care about this topic, please read this great google paper http://googlewebmastercentral.blogspot.com/2009/04/research-study-of-sitemaps.html ( april 2009 ) - read the complete paper, not only the blogpost. from the paper ok, basically google struggled with the same question. they do not disclose how they determine value within the sitemap, but ...


5

Sitemaps can save your ass. On one of my sites, I have a large number of links that I prevent search engines from spidering. Long story short, Google was mis-interpreting JS in my forum and triggering lots of 500 and 403 response codes, which I believed were affecting the site's position. I worked around this by excluding the problematic URLs via ...


5

In my experience: with no sitemap submitted, it might take more than 30 days, with sitemap it usually takes a couple of weeks.


5

It is not possible to define any hierarchal structure in your XML sitemap. The XML sitemap is a straight forward list of all your pages. Any hierarchal structure to your pages will be determined by Google when it crawls your site. The hierarchal structure is more relevant to your users. So, your HTML sitemap (if you have one) could be defined in this way. ...


5

Google does appear to index XML sitemaps (like any XML file). It seems that if Google is aware of a URL and it returns a valid response then it's likely to get indexed. Personally, I only submit the sitemap through GWT and include a Sitemap: reference in robots.txt and this is certainly enough to get it indexed. I believe the recommended method to prevent ...


4

I've not run into this myself, but the majority of my projects are applications or sites that otherwise require user accounts so indexing by search engines isn't a focus. That said, I've heard before that SEO has basically rendered sitemaps useless. If you look at the protocol, it's sort of an "honor system" to tell how often a page changes and what the ...


4

No, you shouldn't include # links. They refer to the same page, and the sitemap is saying how often the page changes (as well as page structure). In fact, your server won't get a request for the #. The browser, or crawler bot, requests the URL www.mydomain.com and then when it retrieves the page jumps to the part of the page with the id of comment1, say. ...


4

Yes. It used to be required to have your XML sitemaps in your root directory but it has since been changed to allow you to place your sitemaps within subdirectories of your choosing. From http://www.sitemaps.org/faq.html#faq_sitemap_location Q: Where do I place my Sitemap? It is strongly recommended that you place your Sitemap at the root ...


4

Would generating sitemap xml improve indexing? In theory, but there are no promises, either. ("Google doesn't guarantee that we'll crawl or index all of your URLs." etc.) Sitemaps can be helpful for bots to find all your pages, but they're informational; they aren't required to do anything with them. The search engines may decide to disregard some ...


4

Yes, submit those pages. They may be empty right now, but they may not be empty when Google crawls them or when a user clicks on the link in the search results. Since the point of the XML Sitemap is to let Google know the page is there, not what the content is, then you should be including those pages.


4

There is no timetable for when Google will crawl and/or index your pages. For new sites it usually is a not very quick process unless you happen to get a high quality link or two pointing to your pages. That always speeds things along but rarely happens. The best thing you can do to speed up the process is to promote your site and seek realted/quality links ...


4

It's depends on which search engine you refer to. Most people are interested in google so I'll assume that's what you meant: Generally speaking, the higher your rank is, and if the frequency in which you change your site (add articles etc) is high - your site will be crawled more frequently. The following article might interest you.



Only top voted, non community-wiki answers of a minimum length are eligible