1

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. Today I am getting today a minimum of 1000 new profile pages today and 10-40 pages getting indexed by Google in 24 hours. That is fine.

Here is my problem:

But already indexed pages are being again and again crawled by Google after some period of time. But that's actually not required and the site is now having 300000 - 400000 pages.

So I don't want the search engine to be crawled for old profile pages again and again unless until there is new update and it is fine to crawl the new pages. Also i already using 410 redirection for the expired profile pages.

It would be great if you suggest, so that Search engine focus for new profile pages only instead of old pages.

Improve this question
3
  • How many pages does your site normally? Can each profile page be seen by more than one URL? Are you using a sitemap?
    – closetnoc
    Commented Aug 8, 2015 at 23:26
  • @closetnoc, Are you using a sitemap? - No. i don't use sitemap. Can each profile page be seen by more than one URL? - Yes. How many pages does your site normally? - Its based upon the user registration. Currently i am getting 80 to 90 registration per day and i could see 10 to 20 indexed pages in the google with 24 settings.
    – Raja
    Commented Aug 10, 2015 at 15:29
  • 1
    The next thing I would tell you is to use the canonical tag per profile page that points to one primary use of the profile page. You do this so that Google knows that they are all the same page. In this case, Google should only index the one primary profile page. Doing this, in time, Google should drop all the other instances of the profile page from the index and the SERPs as the pages are refetched. This link from Google should help: support.google.com/webmasters/answer/139066?hl=en
    – closetnoc
    Commented Aug 10, 2015 at 15:55

2 Answers 2

Reset to default
1

If you haven't already done so, register an account with Google Webmaster Tools and add your domain to it then access your domain, and select the gear icon and visit Site Settings then you can select "Limit Google's maximum crawl rate" and select a value that you're OK with. Probably the lowest value works well for you which may be 0.002 requests a second (a pause of 500 seconds between requests).

Another thing to try is the "Crawl-delay" directive for robots.txt which allows you to specify the waiting time between two requests from the same server and I think this time is measured in seconds. I'm also not sure if all search engines support it. Here's a link for more info about crawl delay:

https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive

I also recommend continuing to issue HTTP status 410 to URLs you no longer want indexed.

If any other pages on your site link to the old pages, then you may want to add rel="nofollow" to each anchor tag that link to any old page so google wont accidentally try to crawl it.

Improve this answer
1

You may want to consider adding a change-frequency tag to your XML sitemap: http://www.sitemaps.org/protocol.html

Another couple things to look into would be ensuring last-modified headers match the creation (or modification) date of the user's account and, if it works for your application, cache related headers: http://www.mobify.com/blog/beginners-guide-to-http-cache-headers/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.