How to Substitute HTML Entities in RSS?

Question

It happened several times that an automatically generated RSS feed broke due to the presence of HTML entities. This makes sense since RSS is not HTML but the issue is that it is generated from fragments of HTML articles.

As a simplified example, say we extract the first sentence of an HTML article to generate part of an RSS feed and that sentence contains and é (é). Now this appears in the RSS feed and therefore the entire feed is no longer valid. Not only does it it fail validation but most readers abort reading the RSS as they do not recover from the failure.

What is the right thing to do when creating RSS from HTML that includes HTML Entities?

There are a few approach that some to mind such as resolving the entity to a charset but would that be universal? Would an RSS in Unicode instead of ASCII work with readers? Or could the HTML entity definition somehow be imported into RSS? Other ideas are welcome. It should not matter but in this case the generated back-end is PHP.

Answer 1

I think a solution suggested by Tomalak should work universally. Ref: https://stackoverflow.com/questions/3805050/xml-parser-error-entity-not-defined

His solution was to convert character entities to corresponding numerical entities.

You could HTML-parse the text and have it re-escaped with the respective numeric entities only (like:   →  ). In any case — simply using un-sanitized user input is a bad idea.

All of the numeric entities are allowed in XML, only the named ones known from HTML do not work (with the exception of &, ", <, >, ').

I created a simple xml document and tested the solution. In first document I used é and in the second I used the corresponding numeric entity é. The first one returned error while second one was valid.

I. Invalid

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">

<channel>
  <title>Test XML Title</title>
  <description>&eacute;</description>
</channel>

</rss>

II Valid

<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">

<channel>
  <title>Test XML Title</title>
  <description>&#233;</description>
</channel>

</rss>

References:

1. https://www.w3.org/TR/html4/sgml/entities.html

2. https://stackoverflow.com/questions/11176752/converting-named-html-entities-to-numeric-html-entities

Answer 2

RSS (and Atom) are based on XML, so the issue you are having is that the & in the HTML entities isn't valid.

Typically you can solve this by double encoding the entities so that the & becomes &: é.

Most RSS readers will then unencode the first encoding, returning it to the original which is then handled by the HTML viewer.

Alternately, you could update your site and feed to declare that they are using UTF-8 and just go with an é everywhere (you'll still need to encode any actual & characters you use either in copy or url query stings).