Token Map: Introduce an efficient lookup and translation class for string mappings.

[dmsnell]

Trac ticket: Core-60698
(previously Core-60841)

Motivated by the need to properly transform HTML named character references (like &) I found the need for a new semantic class which can efficiently perform search and replacement of a set of static tokens. Existing patterns in the codebase are not sufficient for the HTML need, and I suspect there are other use-cases where this class would help.

---

In #6387 I have built a spec-compliant HTML5 text decoder utilizing this token map. The performance of the new decoder is approximately 20% slower than calling html_entity_decode() directly, except that it properly decodes what PHP can't. In fact, the performance bottleneck in that PR comes from converting into UTF-8 the sequence of digits in numeric character references, not in looking up named character references.

---

This proposal is adding a new class WP_Token_Map providing at least two methods for normal use:

• contains( $token ) returns whether the passed string is in the set.
• read_token( $text, $offset = 0, $skip_bytes ) indicates if the character sequence starting at the given offset in the passed string forms a token in the set, and if so, returns the replacement for that token. It also sets &$skip_bytes to the length of the token so that calling code .

It also provides utility functions for pre-computing these classes, as they are designed for relatively-static cases where the actual code is intended to be generated dynamically, but stay static over time. For example, HTML5 defines the set of named character references and indicates that the list shall not change or be expanded. HTML5 spec. Precomputing can save on the startup-up cost of building the optimized lookup tables.

• WP_Token_Map::from_array( array $mappings ) generates a new token map from the given array of whose keys are tokens and whose values are the replacements.
• to_array() dumps the set of mapping into an array suitable for passing back into from_array().
• WP_Token_Map::from_precomputed_table( ...$table ) instantiates a token set from a precomputed table, skipping the computation for building the table and sorting the tokens.
• precomputed_php_source_table() generates PHP source code which can be loaded with the previous static method for maintenance of the core static token sets.

Other potential uses

• Converting smilies like :)
• Converting emoji sequences like :happy:
• Determining if a given verb/action is allowed in an API call.