2

On my site is a page that hosts all my ad entries. Each URL and its content is different. Google was indexing all the different URLs until entry 4570. As I can see in GWT Google Index they also stopped crawling the new entries (70,000) at this moment.

I would like to understand why Google stopped. I added a sitemap at about this time. The sitemap generator doesn't produce the single ad URLs.

I had to change a page name and put a redirect in my .htaccess file:

Redirect permanent /aerzte/ http://www.example.de/arzt/

Google stopped about one week later to add new URLs. Can one of above be the reason?

Is there a different solution - without sitemap - to get Google to index these URLs?

P.S.: I can add URLs with Google Fetch and these URLs are immediately indexed.

|improve this question

migrated from superuser.com Aug 5 '13 at 18:13

This question came from our site for computer enthusiasts and power users.

2

Without seeing your site, it sounds like you are getting hit with a low-quality content penalty and that's why Google is no longer indexing you. Adding URLs with Fetch will work in the short run but if the penalty is the cause then I would expect you to see those URLs being dropped out over over time.

Other problems may be indicated by the size of your sitemap file (70,000 entries). Individual XML files should be capped at 50,000 entries and if your file contains more than that, Google may be petulantly refusing to index. One way to be sure is to check the server's access log for requests for the site map and see if the Googlebot is still making requests for it.

|improve this answer
1

Create an account at Google Webmaster Tools. In about a week they should be able to tell you exactly where the problem pages for the site are under the status for the domain in question.

They'll say things like "Googlebot couldn't access your site." or "Too many redirects." and they'll also list the 404 errors they receive.

If you've made major changes to your site, then they'll actually queue the site for recrawling and they'll start all over (but it takes a couple of weeks for the changes to be reflected). They'll also indicate their crawl rate (which you should be able to change).

Update

Something most people don't know is that Google will still index your site based on your internal links (and offsite links) even if you don't supply a sitemap. One of my sites is on page one for a search out of 1.8m links and I have no sitemap on it.

|improve this answer
  • have a GWT account and got no messages, I changed the URL Parameters 2 days ago and Google is crawling again. So far nothing has been added to the index though. My sitemap doesnt contain those 76.000 urls yet. Appears to be a server config problem, produces a 500 error when I try to load the page with the 50.000 entries. I am using Arne B.'s GSG plugin 4.0 beta version and it works after I changed the max_exec-time in php.ini on my Xampp. I thought this is a permanent redirect? So I should put 301 in front of it like this? Redirect 301 /aerzte/ example.de/arzt Thanks guys! – user1838185 Aug 6 '13 at 16:58
  • Oh, sorry. The redirect permanent is the same as redirect 301. I normally use mod_rewrite and not mod_alias. If you're getting 500 errors, those will definitely cause an issue. – AbsoluteƵERØ Aug 6 '13 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.