Linked Questions

5
votes
4answers
4k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
2
votes
3answers
312 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
413 views

Sitemap.xml doesn't seem to be working properly [duplicate]

I have a sitemap.xml on one of my domains. It contains the following: <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.domain.com</loc> ...
1
vote
0answers
305 views

Google will render but not index my SPA site [duplicate]

I have a single-page-app / website built with React JS, where Google will only index my index page. All the other URL's in my sitemap are recognised, but either marked with Discovered - currently not ...
1
vote
1answer
132 views

Is it enough to submit an XML file to get Orphan pages indexed and ranked? [duplicate]

We have our sitemap submitted to WMT. But we have a lot of Orphan pages on our website, and it seems that they do not rank or do not appear on search, therefore, hardly have any traffic or no traffic ...
0
votes
1answer
143 views

Would google index a page in sitemap which there is no direct link to it from website? [duplicate]

Let's say I have a listing website with thousands of pages. I don't want to make all those pages(items) crawlable by others, unless they search for those items. But, still I like to have all those ...
0
votes
1answer
93 views

Are internal links to pages more valuable than sitemap.xml links [duplicate]

So part of my site is asynchronously loaded with javascript. This part contains links to products. All these links also exist in my sitemap.xml. I've tested with Googlebot Fetch and it doesnt see ...
12
votes
7answers
8k views

How to get tens of millions of pages indexed by Google bot?

We are currently developing a site that currently has 8 million unique pages that will grow to about 20 million right away, and eventually to about 50 million or more. Before you criticize... Yes, it ...
27
votes
5answers
4k views

Page appears indexed in Google but not findable for any search terms?

(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.) If you do a Google search for uiviewcontroller best practices either with ...
7
votes
2answers
2k views

Make Google index over 90,000 user profiles

I have a large amount of user profiles hosted in this schema https://example.com/stats.php?player=name (It's not the prettiest URL, but .htaccess dramatically slows down my site and I don't understand ...
8
votes
2answers
9k views

Is it necessary to submit sitemap to www and non www?

I have submitted my website to Google Webmaster Tools with both www and with no-www. I had made the domain without www to my preffered domain and have submitted a sitemap. My question is should I ...
13
votes
1answer
2k views

Should I include everything in the sitemap or only new content?

For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the ...
5
votes
3answers
2k views

Should I include forum posts in the sitemap?

I am launching a medical discussion forum for sharing study materials and discussing other topics. The forum will probably grow fast and reach at least a few hundred topics. Should every discussion ...
6
votes
3answers
3k views

How long does it take for Google Webmasters to index site after submiting sitemap? [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have submitted my website's sitemap to webmaster's tools. The status on the sitemap says OK and it shows that 12 URLs ...
4
votes
3answers
2k views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my page,...

15 30 50 per page