4
votes
4answers
3k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
0
votes
3answers
244 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
207 views

Sitemap.xml doesn't seem to be working properly [duplicate]

I have a sitemap.xml on one of my domains. It contains the following: <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.domain.com</loc> ...
25
votes
5answers
3k views

Page appears indexed in Google but not findable for any search terms?

(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.) If you do a Google search for uiviewcontroller best practices either with ...
7
votes
4answers
4k views

How to get tens of millions of pages indexed by Google bot?

We are currently developing a site that currently has 8 million unique pages that will grow to about 20 million right away, and eventually to about 50 million or more. Before you criticize... Yes, it ...
7
votes
2answers
1k views

Make Google index over 90,000 user profiles

I have a large amount of user profiles hosted in this schema https://example.com/stats.php?player=name (It's not the prettiest URL, but .htaccess dramatically slows down my site and I don't understand ...
13
votes
1answer
2k views

Should I include everything in the sitemap or only new content?

For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the ...
5
votes
3answers
1k views

Should I include forum posts in the sitemap?

I am launching a medical discussion forum for sharing study materials and discussing other topics. The forum will probably grow fast and reach at least a few hundred topics. Should every discussion ...
6
votes
2answers
3k views

Is it necessary to submit sitemap to www and non www?

I have submitted my website to Google Webmaster Tools with both www and with no-www. I had made the domain without www to my preffered domain and have submitted a sitemap. My question is should I ...
3
votes
4answers
4k views

Google search does not show sub-pages from my website

My website appears in Google search, but only the first page. Of course I have sub-pages linked from the first page, but the sub-pages do not show in Google search. Not in Yahoo, not in Bing. What ...
3
votes
4answers
668 views

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...
7
votes
3answers
962 views

How soon Google is supposed to include new pages to search

I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How soon after Googlebot fetches new sitemap new entities should be ...
6
votes
3answers
433 views

Is submitting a sitemap an outdated technique for SEO?

I'm not getting more content or recommendations for sitemap practice in SEO. Is submitting a sitemap an outdated practice in SEO or is it necessary to submit a sitemap for SERP?
3
votes
3answers
797 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my page,...
7
votes
6answers
279 views

In regards to SEO, is a site map important for a small 3 page website?

In regards to SEO, is a site map important for a small 3 page website? I understand that it is not needed for the convenience of a user visiting the site, but will it help the search engines find it?

15 30 50 per page