We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [sitemap]

A site map (or sitemap) is a list of pages of a web site accessible to crawlers or users.

Filter by
Sorted by
Tagged with
251
votes
19answers
37k views

The Sitemap Paradox

We use a sitemap on Stack Overflow, but I have mixed feelings about it. Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow ...
9
votes
4answers
14k views

Sitemaps - one per subdomain or one for the base domain?

I have more than 5 subdomains on a project, say developer, www, staff and some more, what is the best practice in creating sitemaps? All subdomains that share a same sitemap at base domain (example....
9
votes
4answers
6k views

Google doesn't seem to update the description or title of my homepage

Before we launched our website, we had set up a "coming soon" page and google picked up the title and description from its contents. So the description in the search results said, "Coming soon! Visit ...
5
votes
6answers
4k views

How to tell Google that I have changed my website URLs?

I have done major updates to my website, and renamed all my URLs. How can I tell Google that I have renamed these URLs and let Google refresh its index? My website URL: http://www.pndmasr.com/ My ...
23
votes
4answers
10k views

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search ...
3
votes
1answer
5k views

How to fix “404 Not Found” crawl errors reported by Google in Search Console

I was checking on Google webmaster tools for my blog site to see if there's any indication on why my blog traffic decreased to half in one day and i saw 43 Not Found crawl errors and 5 in Sitemap Not ...
5
votes
6answers
733 views

Looking for a sitemap generator

I am looking for a good way to generate a sitemap for my current site. I inherited a huge site at work and we are going to start the process of IA analysis and redesign. It would be awesome if there ...
3
votes
3answers
1k views

HTTP to HTTPS: Wait for new sitemap to be indexed?

Running HTTP and HTTPS versions of the site side-by-side. HTTP is canonical right now. Goal is to change that to HTTPS. Created a new site in Search Console and submitted a new sitemap 4 days ago. ...
9
votes
7answers
8k views

Does Google penalize daily updated <lastmod> tags in sitemaps if the data is not daily updated?

I've got a sitemap that is generated daily with a lot of links to product pages. These products are imported daily from another data source. Because the update consists of throwing away all current ...
4
votes
1answer
48 views

SEO consequences for category pages 404-ing

Question, is it ok for one of the pages levels to 404? (SEO-wise) Let's say I have a sitemap like this: example.com --/services --/foo --/bar --/sectors --/my-sector --/another-sector --/...
73
votes
5answers
30k views

Are there any clear indicators that my sitemap file is beneficial?

I have recently created a sitemap.xml file and uploaded it to my Google Webmasters Tools account. Google didn't report any issues or errors with the uploaded sitemap of my site. Now my question is: ...
10
votes
1answer
4k views

Can I host a sitemap on another domain?

Just out of curiosity, I have developed a small app analysing and generating sitemaps for a given domain. Can I tell search engines to look for the sitemap of a given domain on my app using a ...
4
votes
1answer
637 views

Should paginated sites be included in sitemap?

My website is ad listing for boats, boats' engines, boat rentals, etc (www.mynautics.com). There are ads listed on multiple pages. I have finished (not jet seen on link above) pagination: added prev ...
2
votes
4answers
201 views

How to remove indexed pages from google effectively

My site was getting traffic and it was on the rise, then I added https and somewhere in that time I also created tons of extra link for google to crawl, thinking this will increase traffic in those ...
7
votes
3answers
8k views

How to submit sitemap to Baidu?

I've some pages in my site in Chinese but I don't understand Chinese (it's some GIS data). Is there any way to submit sitemap to Baidu?
11
votes
4answers
3k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
7
votes
1answer
274 views

If a website is accessed on HTTPS instead of HTTP, should the URLs on the sitemap be HTTPS or HTTP?

I have changed my website from HTTP to HTTPS. At this moment, I have the next rewrite rule working to redirect calls for HTTP to HTTPS: RewriteEngine On RewriteCond %{SERVER_PORT} 80 RewriteRule ^(...
7
votes
2answers
1k views

Are there specific advantages or disadvantages of an XML sitemap over a TXT based sitemap?

Are there any specific advantages or disadvantages in generating an XML based sitemap in place of a simple txt based sitemap (list of URLs)? I realize that in the XML format I can set priority and ...
6
votes
4answers
6k views

Google Webmaster Tools says my XML sitemap “appears to be an HTML page”

We're running a lot of sites and we've started to get a lot of these errors in Webmaster Tools: Sitemap is HTML Your Sitemap appears to be an HTML page. Please use a supported sitemap format ...
4
votes
1answer
2k views

Submitting a URL to Google using PHP/cUrl

I've been successfully submitting xml sitemaps to Google using cURL. The URL is http://www.google.com/webmasters/sitemaps/ping?sitemap= The problem is that XML sitemaps take several days to be ...
3
votes
2answers
6k views

Free unlimited-page sitemap generator? [duplicate]

Possible Duplicate: Looking for a sitemap generator I have been using www.xml-sitemaps.com/ but their limit is 500 pages. My site has grown beyond that and I am wondering whether there is ...
3
votes
1answer
601 views

Does name of sitemap file that Wordpress generates matter?

The Yoast SEO plugin for wordpress creates an XML sitemap with the file name sitemap_index.xml. There is another sitemap plugin (Google XML Sitemaps) in the Wordpress site I'm working on that creates ...
3
votes
1answer
2k views

Should I change the URLs in the XML sitemap after 301 redirecting them?

I am using 301 redirects for all URLs. Should I change sitemap URLs now? The sitemap currently contains the following URLs: example.com/cat/1.html example.com/cat/2.html example.com/cat/3.html ...
1
vote
2answers
752 views

Do Google sitemap URLs need to be double urlencoded?

Do Google sitemap URLs need to be double urlencoded? I am submitting my sitemaps to Google and Google is having difficulty navigating to URLs that contain urlencoded characters. An example of a ...
1
vote
1answer
5k views

How to control what appears on google search results?

So I have just built a site and all is well... but I want to control what appears on google search results. Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
1
vote
1answer
605 views

Sitemap update lastmod and ping. what is the advantage?

My website has 6milions pages of companies. ex: www.mywebsite.com/company-a I will generate one sitemap index file and put the url in robots.txt When the company A update one information in your ...
0
votes
2answers
496 views

Question about paginated based URLs and category URLs with XML sitemaps

I have a blog site that has this URL structure: http://website.com/category/ http://website.com/category/page1/ http://website.com/category/subcat/ http://website.com/category/article_title A flow ...
7
votes
4answers
214 views

Sitemaps HTML and others [closed]

I am looking for the best resources on how to build sitemaps. I am currently aware of the obvious HTML sitemaps and the (google) sitemap.xml and (yahoo) urllist.txt Anyone know of resources for ...
7
votes
2answers
1k views

Is it possible to make an XML sitemap pretty enough to show to users?

I wouldn't mind showing my XML sitemap to users if there were a way to make it human usable as opposed to just machine readable. Is it possible to: Choose the colors and layout Make the locations ...
4
votes
2answers
3k views

Can a sitemap index file list sitemaps in a different directory?

If I have a sitemap index, can the sitemaps listed in the index be located in a subdirectory, but have links that isn't specific to the subdirectory? I'm trying to avoid having multiple sitemaps in ...
14
votes
4answers
8k views

How long does Bingbot typically take to process a sitemap submission?

I submitted a domain's sitemap.xml to Bingbot (in Webmaster Tools). It's been a couple of days but the status is still pending. Does anyone know how much time does Bing Webmaster Tools takes to ...
10
votes
2answers
841 views

Can an XML sitemap file by generated dynamically, or does it have to be static?

Does it matter if I serve sitemap.xml as a dynamic page? e.g., use .htaccess to rewrite sitemap.xml (or whatever static URL I submitted at the Webmaster Tools) to a PHP file. The PHP file will call ...
9
votes
1answer
619 views

Internal links to pages vs. sitemap.xml links to pages

I have a page that is not linked to from anywhere, but it is listed in sitemap.xml. It will be crawled and display in the SERP. Would it be preferable, from a SEO perspective, to have the page ...
7
votes
3answers
1k views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
6
votes
1answer
788 views

How to tell bots to forget a website and reindex it from scratch

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch? Context: After replacing a ...
6
votes
1answer
2k views

Flat vs. Hierarchical Tree Structure

I am a developer in the middle of the development of a medium-sized website. A requirement that we just received from the resident SEO person on the project is for all the URLs on the site to be just ...
6
votes
3answers
5k views

Sitemap: Hierarchical layout possible?

Is it possible to design my sitemap hierarchically? My site is laid out hierarchically into World=>Continent=>Country=>State=>City=>Articles Can I design my sitemap xml like this?...
5
votes
3answers
8k views

Sitemap to display hash tag links or not?

I am just creating sitemap for my site and wanted to find out if I should include # links for SEO? For Example: www.mydomain.com/#comment1 www.mydomain.com/#comment2 Does this help with SEO or ...
4
votes
1answer
2k views

Is it necessary to re-submit modified sitemaps?

Assuming a sitemap.xml has been registered properly to major players, do these players come and revisit those sitemap.xml from time to time to check for updates and modifications (say new entries are ...
4
votes
2answers
654 views

sitemap that contains millions of links/URLs

I am currently working on a shopping / auction website. The involving party who's doing SEO consultancy is insisting to generate URLs for every single combination of products listed on the website, ...
4
votes
2answers
4k views

Should 404 pages feature in your sitemap.xml

I tend to generate the sitemaps for my static sites (no CMS) by using a service such as xml-sitemaps.com. They way this works if you input your homepage and it crawls all the internal links of a site ...
4
votes
1answer
1k views

How to have 2 sitemaps - 1 for main site the other for wordpress in subfolder

Background My site runs on a dedicated server and and has Wordpress in a subfolder. I have my mainsite which is not wordpress and that has a sitemap - for example example.com/sitemap.xml ... that ...
4
votes
1answer
10k views

How to add sitemap to my Blogspot blog? [closed]

I added a lot of pages in my blog. I need to track the blog traffic. If anyone know about adding sitemap to my blog, please send step by step tutorial for this process.
4
votes
1answer
123 views

Moving folder's content to a new domain, what SEO steps to take?

So, we have a functionality on example.com/folder that we decided to make as separate site and we're in need of moving it to a new domain (brand it)... This is what I'm going to make SEO wise (...
4
votes
1answer
495 views

Determining which sitemap entries not indexed by google

We submit a sitemap to google and can see this being indexed in webmaster tools. We have 5140 entries, and have broken this up to 10 child site maps of 500 each. This all seems to be working well. ...
3
votes
2answers
2k views

Google Search Console only showing one page in index status, while many pages indexed in sitemap

I am using Google webmaster tools for my site. I recently have changed address. Now, I wonder why the number of indexed pages in Crawl -> Sitemaps and Google Index -> Index Status are different?
3
votes
2answers
256 views

How should you inform search engines of the location of an XML sitemap?

There are 2 ways to make search engines aware of a sitemap's location that I know of: Include an absolute link to it in robots.txt Submit it to them directly. The relevant URLs are: http://www....
3
votes
3answers
2k views

Where can I find a list of search engine crawler user agents and their domain names? [closed]

I'm trying to follow stackoverflow's approach of using a white list for the sitemap. I have tried to find an extensive list of search engine crawlers (and other crawlers) to use in my white list but ...
3
votes
1answer
391 views

How should I approach sitemap.xml, hreflang and regions for my website

I am a developer on a large site which regionally targets some content with a specific url. Is it technically correct to target the countries in those regions in the sitemap.xml and point them to the ...
2
votes
1answer
100 views

Correctly handling multi language URL's in sitemaps

I have an English page: https://www.example.com/content/article-1 This page is also avaliable in other languages at the URLs: https://www.example.com/de/content/article-1 https://www.example.com/zh-...