Tagged Questions
1
vote
2answers
134 views
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.
I decreased crawl rate from Google webmaster tools.
But I'm supposing to ...
4
votes
3answers
553 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
0
votes
1answer
80 views
Specifying crawlers to not crawl Links which are dependent on external API's
I have links on my site which lead to internal pages that depend on first bringing data from an external API. This only takes time the first time they are pushed. Links to pages which exists in the ...
