1

I have a directory website - I didn't want to have an id number in a URL structure so I chose to store the URLs in the database table.

For example: if http://www.example.com/my-unique-url-without-identifier/ is visited, I search for my-unique-url-without-identifier key in the database table and then put the result based on that.

Now, some of my URL elements have special characters (single quotes, brackets etc) - I encoded them before saving into the database table.

For example: if I have to form the link for king's landing, it is saved as /king%27s-landing/ (instead of /king's-landing/) and all the links across the site are formed using this encoded structure.

However, google's search console is reporting high number of crawling errors and all the pages are of the type where URL is encoded with special characters (apostrophe, brackets etc). Google is somehow trying to crawl http://example.com/king's-landing/ and getting 404, which is correct behavior because I don't have /king's-landing/ in db tables. I am very sure, I only link to /king%27s-landing/ on the website and also included only encoded urls in sitemaps. Also, I have checked the "Linked from" pages and the URLs are encoded urls.

How do I resolve this problem? Is this a known bug with search console?

  • Can't you create links to /kings-landing/ instead (without ')? – Kenny Lajara Jun 21 '16 at 14:28
1

I create URLs like this automatically daily. I scrub non-alpha-numeric characters in the process. I would avoid characters that require encoding in the URL and link. It is not necessary. Search engines can figure out what you mean especially if you specify it correctly within content.

My advice would be to fix your URLs and links so that these special characters do not exist. As for existing URLs that the search engines already have, I would just let them 404 or present a 410 if you want to do the work and just let the process work itself out. It will take a while, however, you will better off for it in the end.

  • Thanks for the suggestion. I never witnessed problems using encoded URLs earlier but in all those places, I used numeric id within URLs and thus it works even when invoked with special characters. I've removed all special characters from the URLs and since they are a small percentage of total number of pages, it should sort out over time. – Aakash Jun 24 '16 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.