Linked Questions

4
votes
4answers
4k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
1
vote
3answers
327 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
0
votes
1answer
427 views

Sitemap.xml doesn't seem to be working properly [duplicate]

I have a sitemap.xml on one of my domains. It contains the following: <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.domain.com</loc> ...
1
vote
0answers
405 views

Google will render but not index my SPA site [duplicate]

I have a single-page-app / website built with React JS, where Google will only index my index page. All the other URL's in my sitemap are recognised, but either marked with Discovered - currently not ...
0
votes
1answer
205 views

Would google index a page in sitemap which there is no direct link to it from website? [duplicate]

Let's say I have a listing website with thousands of pages. I don't want to make all those pages(items) crawlable by others, unless they search for those items. But, still I like to have all those ...
1
vote
1answer
156 views

Is it enough to submit an XML file to get Orphan pages indexed and ranked? [duplicate]

We have our sitemap submitted to WMT. But we have a lot of Orphan pages on our website, and it seems that they do not rank or do not appear on search, therefore, hardly have any traffic or no traffic ...
0
votes
1answer
99 views

Are internal links to pages more valuable than sitemap.xml links [duplicate]

So part of my site is asynchronously loaded with javascript. This part contains links to products. All these links also exist in my sitemap.xml. I've tested with Googlebot Fetch and it doesnt see ...
0
votes
1answer
16 views

Would google index our page if we have it in sitemap but do not have any links for that on website? [duplicate]

Short Would google index my web-page if it's not on my website but it's in the sitemap? Long We have a few product price pages i.e. for product bwm-x3, following are 3 URLs: -> bmw-x3-price-in-...
0
votes
0answers
8 views

XML Sitemap submitted but few page show indexed via Google site: search [duplicate]

I submitted a sitemap of my website to Google. (japan1900.com) The sitemap : (https://japan1900.com/en/module/lgsitemaps/sitemap?fc=module&name=product_1_1) The robots.txt : https://japan1900.com/...
13
votes
6answers
10k views

How to get tens of millions of pages indexed by Google bot?

We are currently developing a site that currently has 8 million unique pages that will grow to about 20 million right away, and eventually to about 50 million or more. Before you criticize... Yes, it ...
14
votes
3answers
5k views

Google Search Console is making up URLs which don't exist in my Sitemap and then complains that these pages have error

I am working on an e-commerce website. I used to have a sitemap with category Ids, something like: <url> <loc>https://my-domain.com/home/browse/2/45/139</loc> <changefreq&...
28
votes
5answers
4k views

Page appears indexed in Google but not findable for any search terms?

(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.) If you do a Google search for uiviewcontroller best practices either with ...
9
votes
2answers
11k views

Is it necessary to submit sitemap to www and non www?

I have submitted my website to Google Webmaster Tools with both www and with no-www. I had made the domain without www to my preffered domain and have submitted a sitemap. My question is should I ...
14
votes
1answer
4k views

Should I include everything in the sitemap or only new content?

For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the ...
7
votes
2answers
2k views

Make Google index over 90,000 user profiles

I have a large amount of user profiles hosted in this schema https://example.com/stats.php?player=name (It's not the prettiest URL, but .htaccess dramatically slows down my site and I don't understand ...

15 30 50 per page
1
2 3 4 5
11