Linked Questions

3
votes
4answers
2k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
0
votes
3answers
188 views

How to make Google index all sitemap.xml records? [duplicate]

I have a sitemap.xml index fine with 3 XML files in it with around 120 pages for indexing. A year passed and I still have 1/3 of it indexed. All the most important pages for indexing are in 1st ...
1
vote
1answer
141 views

Sitemap.xml doesn't seem to be working properly [duplicate]

I have a sitemap.xml on one of my domains. It contains the following: <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.domain.com</loc> ...
24
votes
5answers
3k views

Page appears indexed in Google but not findable for any search terms?

(Note that I am going to use screenshots here because I suspect writing about this will change the behavior over time.) If you do a Google search for uiviewcontroller best practices either with ...
7
votes
4answers
3k views

How to get tens of millions of pages indexed by Google bot?

We are currently developing a site that currently has 8 million unique pages that will grow to about 20 million right away, and eventually to about 50 million or more. Before you criticize... Yes, it ...
7
votes
2answers
1k views

Make Google index over 90,000 user profiles

I have a large amount of user profiles hosted in this schema https://example.com/stats.php?player=name (It's not the prettiest URL, but .htaccess dramatically slows down my site and I don't understand ...
13
votes
1answer
2k views

Should I include everything in the sitemap or only new content?

For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the ...
3
votes
5answers
3k views

Google search does not show sub-pages from my website

My website appears in Google search, but only the first page. Of course I have sub-pages linked from the first page, but the sub-pages do not show in Google search. Not in Yahoo, not in Bing. What ...
5
votes
3answers
647 views

Should I include forum posts in the sitemap?

I am launching a medical discussion forum for sharing study materials and discussing other topics. The forum will probably grow fast and reach at least a few hundred topics. Should every discussion ...
3
votes
4answers
488 views

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...
7
votes
6answers
254 views

In regards to SEO, is a site map important for a small 3 page website?

In regards to SEO, is a site map important for a small 3 page website? I understand that it is not needed for the convenience of a user visiting the site, but will it help the search engines find it?
5
votes
2answers
1k views

Is it necessary to submit sitemap to www and non www?

I have submitted my website to Google Webmaster Tools with both www and with no-www. I had made the domain without www to my preffered domain and have submitted a sitemap. My question is should I ...
5
votes
3answers
331 views

Is submitting a sitemap an outdated technique for SEO?

I'm not getting more content or recommendations for sitemap practice in SEO. Is submitting a sitemap an outdated practice in SEO or is it necessary to submit a sitemap for SERP?
7
votes
2answers
150 views

Can a sitemap change result in a drop of visits?

I recently restructured and resubmitted the sitemap for a website of mine in GWT. The old one was built by a script I acquired, which had its flaws but dit the job. After having implemented URL ...
3
votes
3answers
356 views

How to make Google index millions of user profile pages?

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind: 1- There is no user listing on my ...

15 30 50 per page