3

Does an XML sitemap have to be linked to from "normal" site pages? Are there standard sitemap name and location where bots look for it so that I wouldn't need to link to it? Can I just put it in document root and give it some standard name?

Does it completely depend on particular web crawler?

  • Hi @olegst, Sitemaps are a great resource, not only for search engines but also for users. Depending on how large your website may be, you might want to make your sitemap available for your users via the site's main or secondary navigation - so they can see your entire site at a glance. It's really helpful, and many corporate and government sites do this. – NDEIGU Apr 23 '15 at 11:52
  • 1
    I've never seen an XML sitemap available to users. They generally are not formatted in a way that is useful to users. I suppose it could be possible to provide an XSLT stylesheet to make it pretty. Actually, that seems like a pretty good question in and of itself, so I asked it: webmasters.stackexchange.com/questions/79525/… – Stephen Ostermiller Apr 23 '15 at 12:05
1

From the documentation:

Informing search engine crawlers

Once you have created the Sitemap file and placed it on your webserver, you need to inform the search engines that support this protocol of its location. You can do this by:

  • submitting it to them via the search engine's submission interface
  • specifying the location in your site's robots.txt file
  • sending an HTTP request

The search engines can then retrieve your Sitemap and make the URLs available to their crawlers.

Submitting your Sitemap via the search engine's submission interface

To submit your Sitemap directly to a search engine, which will enable you to receive status information and any processing errors, refer to each search engine's documentation.

Specifying the Sitemap location in your robots.txt file

You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line including the full URL to the sitemap:

Sitemap: http://www.example.com/sitemap.xml

This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file. You don't need to list each individual Sitemap listed in the index file.

You can specify more than one Sitemap file per robots.txt file.

Sitemap: http://www.example.com/sitemap-host1.xml

Sitemap: http://www.example.com/sitemap-host2.xml

Submitting your Sitemap via an HTTP request

To submit your Sitemap using an HTTP request (replace with the URL provided by the search engine), issue your request to the following URL: /ping?sitemap=sitemap_url

For example, if your Sitemap is located at http://www.example.com/sitemap.gz, your URL will become:

<searchengine_URL>/ping?sitemap=http://www.example.com/sitemap.gz

URL encode everything after the /ping?sitemap=:

<searchengine_URL>/ping?sitemap=http%3A%2F%2Fwww.yoursite.com%2Fsitemap.gz

You can issue the HTTP request using wget, curl, or another mechanism of your choosing. A successful request will return an HTTP 200 response code; if you receive a different response, you should resubmit your request. The HTTP 200 response code only indicates that the search engine has received your Sitemap, not that the Sitemap itself or the URLs contained in it were valid. An easy way to do this is to set up an automated job to generate and submit Sitemaps on a regular basis.

Note: If you are providing a Sitemap index file, you only need to issue one HTTP request that includes the location of the Sitemap index file; you do not need to issue individual requests for each Sitemap listed in the index.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.