Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I previously used to run my website on WordPress but now I switched to a plain HTML and CSS template.
However, wp-includes folders are indexed in Google search which have no meaning now, and Google Search Console keeps displaying an error:
Indexed, though blocked by robots.txt
Obviously there is nothing to do now to those folders. Should I remove these results one-by-one, or there is there another approach for this?
I believe you have physically removed the wp-includes folder so that server returns 404 for the concerned URLs. If yes, you should not block the URLs in robots. Else Google will not try crawling those URLs. Only if Google tries to crawl, it will know if the URLs have been removed. Only then it will start removing them from the index.
Crawl and index instructions are very literal. I see it all the time where people try to use one type to get another type's effect, when they really need that type. For instance what you want is pages removed from the index, but what you're saying is "don't crawl my pages".
Those things are not the same thing.
If you block with robots.txt, Google will attempt to infer information about the page from the links going to that page. So for instance if NFL.com set their entire site to Robots Noindex, Google would infer all the popular pages on the site, solely based on links from other sites into NFL.com. In fact, once I checked it and found the word "football" did not exist anywhere on the page. Yet it was #1 for that term. Why? Third party links.
So Google will infer the existence of those pages based on outside links... robots.txt will actually backfire! Since Google is barred from physically fetching the pages off your site, it will never know they are 404! And will continue to index them indefinitely, using link data alone.
Your second choice of what to do there is to let Google crawl, but 404 it. Your best choice is a HTTP 301 (permanent) redirect, or an HTML meta-tag "refresh" redirect if you can't create HTTP 301 redirects.
Don't abandon URLs. Lots and lots of URLs are either bookmarked or are human-curated links to your page from other pages.
I know it's a chore, but don't let those go 404. Every link that visitors get to "by bookmark or link" should redirect to the current location of that specific content. DO NOT punt them to the homepage. Send them to the new location of that content page.
The good news is, you don't need to do every link; just the ones that pop up in server logs as 404 (that aren't from robots). My rule of thumb is this is necessary for pages which come up as 404 in server logs (from users not bots). That means there is human interest in the page, and they need to be sent to the right content.
Google uses links from other sites to yours as a ranking factor. It may also use Chrome users' bookmarks and past activity for the same purpose. When you use a proper redirect, Google transfers that ranking factor ("link juice") from your old URL to your new one. (internal to the site, that is).
Installing a new CMS and just neglecting all those 404s is a klutzy thing to do, that hurts your site's reputation and ranking.
Everybody wants to use the latest popular CMS. One of my criteria for a CMS is it must respect my existing URL structure - that is, if the CMS requires me to move all my URLs, I won't use it. One should change URLs very reluctantly, and only as a last resort. If your web designer disagrees, that opinion is for the web designer's convenience at the expense of your customer relations and search engine traffic.
