New answers tagged robots.txt
1
vote
Googlebot is crawling unpublished pages from Adobe AEM and getting 404 errors
If the unpublished pages are in your sitemap, then removing them from your sitemap is the ideal solution.
Write robots.txt rule for each page affected to be ignored?
This is likely to work. But ...
- 364
Top 50 recent answers are included