Crawling errors due to encoded URLs - urls are stored in DB table

Question

I have a directory website - I didn't want to have an id number in a URL structure so I chose to store the URLs in the database table.

For example: if http://www.example.com/my-unique-url-without-identifier/ is visited, I search for my-unique-url-without-identifier key in the database table and then put the result based on that.

Now, some of my URL elements have special characters (single quotes, brackets etc) - I encoded them before saving into the database table.

For example: if I have to form the link for king's landing, it is saved as /king%27s-landing/ (instead of /king's-landing/) and all the links across the site are formed using this encoded structure.

However, google's search console is reporting high number of crawling errors and all the pages are of the type where URL is encoded with special characters (apostrophe, brackets etc). Google is somehow trying to crawl http://example.com/king's-landing/ and getting 404, which is correct behavior because I don't have /king's-landing/ in db tables. I am very sure, I only link to /king%27s-landing/ on the website and also included only encoded urls in sitemaps. Also, I have checked the "Linked from" pages and the URLs are encoded urls.

How do I resolve this problem? Is this a known bug with search console?

Answer 1

I create URLs like this automatically daily. I scrub non-alpha-numeric characters in the process. I would avoid characters that require encoding in the URL and link. It is not necessary. Search engines can figure out what you mean especially if you specify it correctly within content.

My advice would be to fix your URLs and links so that these special characters do not exist. As for existing URLs that the search engines already have, I would just let them 404 or present a 410 if you want to do the work and just let the process work itself out. It will take a while, however, you will better off for it in the end.