New answers tagged googlebot
1
check the IP address source in WHOIS. IF it is not Google, then simply block the IP. Also, if the page does not exist, it would be worthwhile to add it, with an action of whatever IP address touches this page is automatically added to the block list for the site.
1
If you are using server side rendering (which doesn't rely on JavaScript), you should be able to use any crawler. The most popular are probably:
wget - free open source, command line based, use the recursive (-r) option
Screaming Frog - requires payment after 500 pages, Windows GUI
Of course there are hundreds of other options if you search for "web ...
1
Have you looked at the Status column? It should show "Success" if Google reads it correctly as well as the last date Google read your sitemap.
Just as an FYI, there's lots of comments here regarding the Sitemap Paradox. It might be beneficial for you to read through that discussion.
0
Server errors (5xx) mean that when the Googlebot tried to crawl those pages, they were unavailable (server was offline, too slow, etc) AT THE TIME MENTIONED; if you tested the live URLs at a later time, it can be that the reason for the server error wasn't present anymore and those pages should be crawled next time Google visits your website (as you ...
0
No, because they are different bots: the shopping googlebot and the web googlebot. Block parameters from the web googlebot, but let open for shopping googlebot. Shopping is the adsbot.
-1
You never know, how certain page will get visitors after creation. Thats why your strategy should be like
Set all new urls on index, follow,
After three months after publishing set all URLs with less then 10 organic visits à day to noindex, follow,
After they are away from index, set them with meta robots to noindex, nofollow and links to them to rel="...
Top 50 recent answers are included
