A site map (or sitemap) is a list of pages of a web site accessible to crawlers or users.

learn more… | top users | synonyms

1
vote
3answers
946 views

Where can I find a list of search engine crawler user agents and their domain names?

I'm trying to follow stackoverflow's approach of using a white list for the sitemap. I have tried to find an extensive list of search engine crawlers (and other crawlers) to use in my white list but ...
1
vote
1answer
45 views

Can externally hosted sitemaps work with Google and Bing?

I am using a "cloud" hosting provider (Heroku) to host my webapp. Since I don't have access to a permanent file system, I am storing my sitemap.xml in Amazon S3. I wanted to know the SEO implications ...
1
vote
1answer
32 views

Custom Sitemap or sitemap generator tool [closed]

I have recently launched a website which is using a free sitemap generator. It's a large travel agency site with predominantly dynamically generated content - users can add their products as and when ...
1
vote
1answer
64 views

Moved sitemaps to a different subdomain and losing search referrals around the same time. Red herring or correlation?

We started to lose search referral traffic around the same time that I moved some of our sitemaps to a subdomain. Could this have hurt us? I followed Google's steps to creating a sitemap under a ...
1
vote
1answer
103 views

How to check if a WordPress SEO plugin is working

I have setup my WordPress site with Yoast. However, it doesn't seem to appear in any of the search engines. In Google Webmaster Tools I don't see a sitemap. Is the plugin supposed to submit the ...
1
vote
1answer
71 views

What to do with discontinued product pages? [duplicate]

I have a site that reviews personal medical devices. Each year, these medical devices get replaced a few times by their newer/improved versions, so the old devices are not available anywhere ...
1
vote
1answer
232 views

Google's webmaster tools saying invalid URL in sitemap

I've recently made some big changes to my site and as a result, have had to change my sitemap. In doing so, I resubmitted my sitemap file only to see 3 errors reading invalid URL's. I'm not sure if ...
1
vote
0answers
231 views

Powermapper alternatives and Google Analytics? [closed]

What is a good application to map the hierarchical structure of a website and maybe get some Google Analytics action going on? Powermapper is pretty expensive and the trial version didn't seem ...
1
vote
1answer
259 views

Displaying thumbnails in google search results for flash games?

Some sites are somehow displaying video preview thumbnails for pages that contain flash games in google search results. For example search for: site:www.thorgaming.com game I see this only on small ...
1
vote
1answer
532 views

Google sitemap access error, something to do with .htaccess redirect from http://www. to http://

I recently updated the htaccess of my site to redirect from http://www. to http://. It all works fine but now I've made some changes to my site and uploaded a new sitemap.xml to google webmaster. I'm ...
1
vote
0answers
368 views

sitemap - following mvcsitemapprovider but categorization is confusing me [closed]

I'm using mvcsitemapprovider for sitemap. http://mvcsitemap.codeplex.com/ public class StoreDetailsDynamicNodeProvider : DynamicNodeProviderBase { MusicStoreEntities storeDB = ...
1
vote
0answers
129 views

How does Stack Overflow have questions appear in Google so quickly? [duplicate]

Possible Duplicate: StackOverflow and Google search I have noticed while asking questions on Stack Overflow that they appear in Google almost instantly. How does SO do this? Is it a feed ...
1
vote
4answers
124 views

How long for Google to index all site maps?

We recently uploaded new sitemaps, as the old ones were out of date and we've also upgraded our forum which changed the URLs of a lot of our website (we're 301 redirecting of course). We uploaded the ...
1
vote
3answers
793 views

Google SEO - remove site map from search result

I am an amateur to SEO. I have a new site recently and submitted the site map to GWT. But after several weeks. I found when I search my domain in google. The site map also appears to be one of the ...
0
votes
3answers
28 views

Sitemap completeness

I have a couple of questions about sitemaps: Does every page need to exists in the sitemap? And what about (ir)relevant parameters on which the content could depend? What about pages which depend ...
0
votes
1answer
355 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
0
votes
1answer
84 views

Google say: robots.txt not valid for subdomains?

I'm very confused about Google's robots.txt documentation Examples of valid robots.txt URLs: Not valid for http://other.example.com/ https://example.com/ http://example.com:8181/ ...
0
votes
1answer
40 views

If my index page has no textual content, should it be present in sitemap?

My index page contains no textual content at all (except some footer links, copyright notice and some stuff in tag). It has, however, big Google's map with some pins attached which lead to articles. ...
0
votes
1answer
77 views

Trade off: Lower the number of URLs in sitemap from 43k to 23k or update the sitemap.xml only weekly basis

we rewrote the sitemap creation process. Now the sitemap contains 43.000 URLs. 20k more than before. We have daily changing in URLs. The script that is creating the complete sitemap takes more than ...
0
votes
3answers
156 views

Google is not indexing the entirety of my site [duplicate]

I require google to index the entirety of my site: 10 or so basic sections with 80,000+ individual job offer pages. However, to date, only 2,900 pages have been indexed by google. I'd like to know ...
0
votes
1answer
111 views

online site map generator that has an access via URL [duplicate]

Possible Duplicate: Looking for a sitemap generator I am looking for a service that I can access daily based to generate a sitemap.xml file for SEO. Something like ...
0
votes
1answer
43 views

What is the optimal way to apply categories to a new site?

To whit: I run a men's lifestyle Q&A site. I have a total of 25 categories in which questions are submitted. Currently, however, these "categories" are in fact sub-categories, such that ...
0
votes
2answers
212 views

General HTTP error: Domain name not found

I have verified my domain through a TXT record by google, however, my site hasn't indexed yet and I face with sitemap errors in google webmaster tool. I run dns health check from ...
0
votes
2answers
101 views

Google Webmaster Tools can cache sitemap files?

I'm struggling to submit my sitemaps to Google Webmaster Tools. Previously I received report that I have an error in all my sitemaps (the error was at <image:image> tag as I did not specified ...
0
votes
1answer
729 views

Sitemap for a one-page site

I'm wondering whether it's worth the effort to create a sitemap for a single-page site. By single page I'm not talking about a webapp made with Angular or Backbone, just a presentation site/page ...
0
votes
1answer
193 views

Is it recommend to create sitemaps with query strings (i.e. URL parameters)?

I am considering creating a "My products" page on a site where a user will be able to search for product information. This page will also display the product information. I need google to know about ...
0
votes
2answers
110 views

How to get Google to display a list of posts in the SERPs for a self hosted WordPress blog?

My website is javatechig.com and is around 6 months old. I have used All-in-One SEO WordPress plugin for SEO tricks. Definitely after using SEO plugin, my website page view is increased, but still it ...
0
votes
2answers
75 views

Trying to create my first sitemap

I'm inserting a website map for the first time. I found this page: http://www.sitemaps.org/protocol.html That page shows examples about sitemaps with xml, should I insert and xml code in my html? ...
0
votes
2answers
205 views

Question about paginated based URLs and category URLs with XML sitemaps

I have a blog site that has this URL structure: http://website.com/category/ http://website.com/category/page1/ http://website.com/category/subcat/ http://website.com/category/article_title A flow ...
0
votes
1answer
1k views

google image sitemap title and caption, which is more important?

here is the document about google image sitemap, But <image:caption> and <image:title>, which is more important? if i have only one description of this image, which part is better to be ...
0
votes
3answers
96 views

Website configuraiton for SEO Optimization

In order to promote our product in the various search engines, we've changed our index.html to redirect to ourapp.com/product/index.html and moved all our HTML code to ourapp.com/product/index.html ...
0
votes
2answers
633 views

Google sitemap for dynamic url structure

I am wondering what the best approach would be to generating a sitemap for google to crawl with a highly dynamic url structure. We have a webapp that assigns custom uris to all the users in our ...
0
votes
1answer
126 views

If Apache applies compression with mod_deflate, is there a reason to serve compressed files?

I have an Apache HTTP server which is configured with mod_deflate to compress content "on the fly". Is there any benefit if I serve a compressed sitemap.xml.gz file, or should I rely on mod_deflate ...
0
votes
1answer
267 views

Code for sitemap — 'weekly.?

I'm creating a Sitemap manually. If I want to specify a url to be crawled weekly, is this the right code: <url> <loc>http://wordfruit.com/</loc> ...
0
votes
1answer
189 views

Sitemap.xml.gz. can I have a compressed file with more than one sitemap in it?

I've got quite a few sitemaps as my site is quite large. Is there any way that I can have a compressed with with multiple sitemap files in it ? I've googled but can't find the answer to this
0
votes
1answer
30 views

Why do sitemaps generated by different plugins look so different from each other?

This is my first time generating a sitemap. I used two plugins to compare sitemaps. Yoast generated a normal looking sitemap. But the sitemap generated by Google XML Sitemaps plugin looks weird (see ...
0
votes
1answer
31 views

Cleanup incorrectly indexed subdomains due to wildcard DNS

I have site www.example.com which someone previously configured with wildcard DNS records and with no 301 redirects. As you may have expected, Google's index is now full of garbage like w.example.com, ...
0
votes
1answer
19 views

What resolution in sitemap.xml image attribute?

I'm just creating a sitemap for my page and I found these aspects for creating it on Google: https://support.google.com/webmasters/answer/183668 There is a attribute for images, but my question is: ...
0
votes
1answer
56 views

Website with over 1 million posts with not much textual content

I've made a website which crawls files from all over the Internet and I feel like Google will ban me if I sent it sitemaps which contain all of these pages (1m+), because they contain only the file ...
0
votes
1answer
98 views

Is there any reason to add pagination to sitemap?

Let's say I have the following site structure: root category1 item1 item2 .... item99 category2 item101 item102 .... item199 ... Each category page contain ...
0
votes
1answer
142 views

Submitting vBulletin Sitemap to Google

I'm using Google Webmaster Tools to get my forum out there, and when I go to submit the sitemap I get the following error: Your Sitemap appears to be an HTML page. Please use a supported ...
0
votes
1answer
62 views

insert multi post page in sitemap or not?

i want to insert below links in Sitemap, Each page has its own unique content : http://poetry.qrosh.com/en/William%20Shakespeare/A%20Fairy%20Song/1/1/ so if i have multi post pages which include 10 ...
0
votes
1answer
21 views

Indexing a page in which the content is discovered in a map

We want to improve our site's SEO. The page is a GWT (javascript based) web page and its content is discovered through a map or a GPS. Now we have just a normal landing html page, and the map and ...
0
votes
1answer
44 views

Google news sitemap to many URLs

I created a Google news sitemap a while ago then about 6 months ago limited it to about 15 news items. The trouble is it seems Google has indexed over a 1000 items and is complaining the sitemap has ...
0
votes
2answers
282 views

Google stopped indexing my site. Is there a substitute for a XML sitemap?

On my site is a page that hosts all my ad entries. Each URL and its content is different. Google was indexing all the different URLs until entry 4570. As I can see in GWT Google Index they also ...
0
votes
1answer
248 views

1000 page submit limit in Google Webmaster Tools sitemap

I'm currently managing Webmaster Tools for some Wordpress sites and I'm curious as to why none of the sites are showing more than 1000 pages submitted. Some of the sites are much larger but I can't ...
0
votes
1answer
102 views

SoLoMo landingpage localized regionalized down to cities and SEO sitemap.xml

I'm working on an SoLoMo (social local mobile) web application. How would you realize the site and the landingpages that they are seo and crawler friendly? Any best-practises out there? I know there ...
0
votes
1answer
135 views

403 Error Crawling Pages

I am receiving more than 6000 errors in Google Webmaster Tools. It is showing "access denied"; can anyone please help me out with resolving this?
0
votes
1answer
194 views

Visual sitemap generater [closed]

I'm looking for a something to visually create a sitemap for one of my websites. Id like something in a tree structure, so I have the hierarchical view of my site. A couple requirements I have ...
0
votes
1answer
245 views

mod_rewrite and SEO friendliness

My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes. I have a unique, large PHP script, and I use the Apache ...