New answers tagged robots.txt
1
vote
Googlebot is crawling unpublished pages from Adobe AEM and getting 404 errors
If the unpublished pages are in your sitemap, then removing them from your sitemap is the ideal solution.
Write robots.txt rule for each page affected to be ignored?
This is likely to work. But ...
Top 50 recent answers are included
Related Tags
robots.txt × 712seo × 211
web-crawlers × 185
google-search-console × 140
google × 89
googlebot × 89
search-engine-indexing × 64
sitemap × 53
google-search × 38
search-engines × 38
noindex × 37
wordpress × 32
google-index × 29
htaccess × 26
xml-sitemap × 26
duplicate-content × 19
subdomain × 15
301-redirect × 14
redirects × 13
url × 13
nofollow × 13
404 × 12
crawl-errors × 12
html × 11
meta-tags × 11