Character Model for the World Wide Web: String Matching

1.1 Goals and Scope

The goal of the Character Model for the World Wide Web is to facilitate use of the Web by all people, regardless of their language, script, writing system, or cultural conventions, in accordance with the W3C goal of universal access. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.

This part of the Character Model for the World Wide Web covers string matching—the process by which a specification or implementation defines whether two string values are the same or different from one another. It describes the ways in which texts that are semantically equivalent can be encoded differently and the impact this has on matching operations important to formal languages (such as those used in the formats and protocols that make up the Web).

The main target audience of this specification is W3C specification developers. This specification and parts of it can be referenced from other W3C specifications and it defines conformance criteria for W3C specifications, as well as other specifications.

Other audiences of this specification include software developers, content developers, and authors of specifications outside the W3C. Software developers and content developers implement and use W3C specifications. This specification defines some conformance criteria for implementations (software) and content that implement and use W3C specifications. It also helps software developers and content developers to understand the character-related provisions in W3C specifications.

The character model described in this specification provides authors of specifications, software developers, and content developers with a common reference for consistent, interoperable text manipulation on the World Wide Web. Working together, these three groups can build a globally accessible Web.

1.2 Structure of this Document

This document defines one of the basic building blocks for the Web related to this problem by defining rules and processes for String Identity Matching in document formats. These rules are designed for the identifiers and structural markup (syntactic content) used in document formats to ensure consistent processing of each and are targeted to Specification writers. This section is targeted to implementers.

This document is divided into two main sections.

The first section lays out the problems involved in string matching; the effects of Unicode and case folding on these problems; and outlines the various issues and normalization mechanisms that might be used to address these issues.

The second section provides requirements and recommendations for string identity matching for use in formal languages, such as many of the document formats defined in W3C Specifications. This primarily is concerned with making the Web functional and providing document authors with consistent results.

1.3 Background

This section provides some historical background on the topics addressed in this specification.

At the core of the character model is the Universal Character Set (UCS), defined jointly by the Unicode Standard [Unicode] and ISO/IEC 10646 [ISO10646]. In this document, Unicode is used as a synonym for the Universal Character Set. A successful character model allows Web documents authored in the world's writing systems, scripts, and languages (and on different platforms) to be exchanged, read, and searched by the Web's users around the world.

The first few chapters of the Unicode Standard [Unicode] provide useful background reading.

For information about the requirements that informed the development of important parts of this specification, see Requirements for String Identity Matching and String Indexing [CHARREQ].

1.4 Terminology and Notation

This section contains terminology and notation specific to this document.

The Web is built on text-based formats and protocols. In order to describe string matching or searching effectively, it is necessary to establish terminology that allows us to talk about the different kinds of text within a given format or protocol, as the requirements and details vary significantly.

A Unicode code point (or "code point") refers to the numeric value assigned to each Unicode character. Unicode code points range from 0 to 0x10FFFF₁₆. (See Section 4.1 in [CHARMOD] for a deeper discussion of character encoding terminology.)

Unicode code points are denoted as U+hhhh, where hhhh is a sequence of at least four, and at most six hexadecimal digits. For example, the character € [U+20AC EURO SIGN] has the code point U+20AC.

Some characters used in this document's examples might not appear as intended on your specific device or display. Usually this is due to lack of a script-specific font installed locally or due to other limitations of your specific rendering system. This document uses a Webfont to provide fallback glyphs for many of the non-Latin characters, but your device might not support displaying the font. To the degree possible, the editors have tried to ensure that the examples nevertheless remain understandable.

A legacy character encoding is a character encoding form that does not encode the full repertoire of characters in the Unicode character set.

A transcoder is a process that converts text between two character encodings. Most commonly in this document it refers to a process that converts from a legacy character encoding to a Unicode encoding form, such as UTF-8.

Syntactic content is any text in a document format or protocol that belongs to the structure of the format or protocol. This definition includes values that are typically thought of as "markup" but can also include other values, such as the name of a field in an HTTP header. Syntactic content consists of all of the characters that form the structure of a format or protocol. For example, < and > (as well as the element name and various attributes they surround) are part of the syntactic content in an HTML document.

Syntactic content usually is defined by a specification or specifications and includes both the defined, reserved keywords for the given protocol or format as well as string tokens and identifiers that are defined by document authors to form the structure of the document (rather than the "content" of the document).

Natural language content refers to the language-bearing content in a document and not to any of the surrounding or embedded syntactic content that form part of the document structure. You can think of it as the actual "content" of the document or the "message" in a given protocol. Note that syntactic content can contain natural language content, such as when an [HTML] img element has an alt attribute containing a description of the image.

A resource, in the context of this document, is a given document, file, or protocol "message" which includes both the natural language content as well as the syntactic content such as identifiers surrounding or containing it. For example, in an HTML document that also has some CSS and a few script tags with embedded JavaScript, the entire HTML document, considered as a file, is a resource. This term is intentionally similar to the term 'resource' as used in [RFC3986], although here the term is applied loosely.

A user-supplied value is unreserved syntactic content in a vocabulary that is assigned by users, as distinct from reserved keywords in a given format or protocol. For example, CSS class names are part of the syntax of a CSS style sheet. They are not reserved keywords, predefined by any CSS specification, but they are still are subject to the syntactic rules of CSS. Users generally expect that keywords they define can be representative of words or concepts in their given natural language, subject to the limitations of those rules.

A vocabulary provides the list of reserved names as well as the set of rules and specifications controlling how user-supplied values (such as identifiers) can be assigned in a format or protocol. This can include restrictions on range, order, or type of characters that can appear in different places. For example, HTML defines the names of its elements and attributes, as well as enumerated attribute values, which defines the "vocabulary" of HTML syntactic content. Another example would be ECMAScript, which restricts the range of characters that can appear at the start or in the body of an identifier or variable name. It applies different rules for other cases, such as to the values of string literals.

A grapheme is a sequence of one or more characters in a visual representation of some text that a typical user would perceive as being a single unit (character). Graphemes are important for a number of text operations such as sorting or text selection, so it is necessary to be able to compute the boundaries between each user-perceived character. Unicode defines the default mechanism for computing graphemes in Unicode Standard Annex #29: Text Segmentation [UAX29] and calls this approximation a grapheme cluster. There are two types of default grapheme cluster defined. Unless otherwise noted, grapheme cluster in this document refers to an extended default grapheme cluster. (A discussion of grapheme clusters is also given in Section 2 of the Unicode Standard, [Unicode]. Cf. near the end of Section 2.11 in version 8.0 of The Unicode Standard)

Because different natural languages have different needs, grapheme clusters can also sometimes require tailoring. For example, a Slovak user might wish to treat the default pair of grapheme clusters "ch" as a single grapheme cluster. Note that the interaction between the language of string content and the end-user's preferences might be complex.

1.5 Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MAY, MUST, MUST NOT, RECOMMENDED, SHOULD, and SHOULD NOT are to be interpreted as described in [RFC2119].

This document describes best practices for the authors of other specifications, as well as recommendations for implementations and content authors. These best practices can also be found in the Internationalization Working Group's document Internationalization Best Practices for Spec Developers [INTERNATIONAL-SPECS], which is intended to serve as a general reference for all Internationalization best practices in W3C specifications.

When a best practice or recommendation appears in this document, it has been styled like this paragraph. Recommendations for specifications and spec authors are preceded by [S]. Recommendations for implementations and software developers are preceeded by [I]. Recommendations for content and content authors are preceeded by [C].

Best practices in this document use [RFC2119] keywords in order to clarify the Internationalization Working Group's intentions regarding a specific recommendation. Following the recommendations in this document can help avoid issues during the W3C's "wide review" process, during implementation, or in the content that authors produce. This document is not, itself, normative and can be revised from time to time.

Specifications can claim conformance to this document if they:

1. do not violate any conformance criteria preceded by [S] where the imperative is MUST or MUST NOT
2. document the reason for any deviation from criteria where the imperative is SHOULD, SHOULD NOT, or RECOMMENDED
3. make it a conformance requirement for implementations to conform to this document
4. make it a conformance requirement for content to conform to this document

Where this specification contains a procedural description, it is to be understood as a way to specify the desired external behavior. Implementations MAY use other means of achieving the same results, as long as observable behavior is not affected.

2.1 Case Mapping and Case Folding

Some scripts and writing systems make a distinction between UPPER, lower, and Title case characters. Most scripts, including the Brahmic scripts of India, the Arabic script, and the scripts used to write Chinese, Japanese, or Korean, do not have a case distinction, but some important ones do. Examples of such scripts include the Latin script used in the majority of this document, as well as scripts such as Greek, Armenian, and Cyrillic.

Case mapping is the process of transforming characters to a specific case, such as UPPER, lower, or Titlecase. For those scripts that have a case distinction, Unicode defines a default UPPER, lower, and Titlecase character mapping for each Unicode code point. Case mapping, at first, appears simple. However there are variations that need to be considered when treating the full range of Unicode in diverse languages.

Case folding is the process of making two texts which differ only in case identical for comparison purposes, that is, it is meant for the purpose of string matching. This is distinct from case mapping, which is primarily meant for display purposes. As with the default case mappings, Unicode defines default case fold mappings ("case folding") for each Unicode code point. Unicode defines two forms of case folding, which we'll examine below.

Since most scripts do not have a case distinction, as with case mappings, most Unicode code points do not require a case folding. For those code points that have a case folding, the majority have a simple, straight-forward mapping to another single matching (generally lowercase) code point. Unicode calls this set of foldings common, since both types of case folding defined by Unicode includes these.

A few characters have a case folding that map one Unicode code point to two or more code points. This set of case foldings are called the full case foldings. The full and common case foldings are used together to provide the default case folding for all of Unicode. We refer to this form of case folding as full casefolding or Unicode full in this document.

Because some applications cannot allocate additional storage when performing a case fold operation, Unicode provides a simple case folding that maps a code point that would normally fold to more or fewer code points to use a single code point for comparison purposes instead. Unlike the full folding, this folding invariably alters the content (and potentially the meaning) of the text. As with full casefolding, the simple casefolding or Unicode simple casefold, is a combination of simple and common mappings so as to cover the full range of Unicode. Unicode simple is not appropriate for use on the Web.

Note that case folding removes information from a string which cannot be recovered later. For example, two s letters in German do not necessarily represent ß in unfolded text.

2.2 Unicode Normalization

A different kind of variation can occur in Unicode text: sometimes several different Unicode code point sequences can be used to represent the same abstract character. When searching or matching text by comparing code points, these variations in encoding cause text values not to match that users expect to be the same.

Because applications need to find the semantic equivalence in texts that use different code point sequences, Unicode defines a means of making two semantically equivalent texts identical: the Unicode Normalization Forms [UAX15].

Resources are often susceptible to the effects of these variations because their specifications and implementations on the Web do not require Unicode Normalization of the text, nor do they take into consideration the string matching algorithms used when processing the syntactic content and natural language content later. For this reason, content developers need to ensure that they have provided a consistent representation in order to avoid problems later.

However, it can be difficult for users to assure that a given resource or set of resources uses a consistent textual representation because the differences are usually not visible when viewed as text. Tools and implementations thus need to consider the difficulties experienced by users when visually or logically equivalent strings that "ought to" match (in the user's mind) are considered to be distinct values. Providing a means for users to see these differences and/or normalize them as appropriate makes it possible for end users to avoid failures that spring from invisible differences in their source documents. For example, the W3C Validator warns when an HTML document is not fully in Unicode Normalization Form C.

2.3 Identical-Appearing Characters and the Limitations of Normalization

Many users are surprised to find that two identical-looking strings—including those that have had a specific Unicode normalization form applied—might not in fact use the same underlying Unicode code points. This includes strings that have had the more-destructive NFKC and NFKD compatibility normalization forms applied to them. Even when strings, tokens, or identifiers appear visually to be the same, they can be encoded differently.

The Unicode canonical normalization forms are concerned with folding the multiple different code point sequences that can be used for a given abstract character or grapheme cluster to use the same code point sequence. However, logically distinct characters or grapheme clusters can still look the same or very similar. When a pair of graphemes look identical (or very similar), they are called homographs. When a pair of graphemes look similar or are homographs but actually represent logically different characters or character sequences, they are said to be confusable.

Examples of identical or identical-seeming appearance can appear even within a single script. This can take the form of similarly shaped characters, such as "0" and "O" or "l" and "1". But other scripts or the use of different compatibility characters can present much less readily distinguished variations. In some cases, Unicode Normalization brings these together, but in many other cases it does not.

Characters that are identical or confusable in appearance can present spoofing and other security risks. This can be true within a single script or for similar characters in separate scripts. For further discussion and examples of homoglyphs and confusability, one useful reference is [UTS39].

In addition to identical or similar-appearing characters, the opposite problem also exists: Unicode Normalization, even the NFKC and NFKD Compatibility forms, does not bring together characters that have the same intrinsic meaning or function but which vary in appearance or usage. For example, U+002E (.) and U+3002 (。) both function as sentence ending punctuation, but the distinction is not removed by normalization because the characters have a distinct identity.

2.4 Interaction of Normalization and Case Folding

One additional complication is that case folding or case mapping can affect the normalization of the string the casing operation is applied to. If the goal is to compare two strings in a case-insensitive manner, the results of a case fold operation have to be renormalized or the resulting case-folded strings might not match.

The Unicode canonical normalization forms and case folding, when used together, are closed: once a string has been case folded and then had NFC or NFD applied to it, further applications of the same case folding and Unicode normalization form do not result in a different string.

When comparing strings while ignoring compatibility variations between characters, an additional level of normalization has to be applied because the compatibility decomposition step can result in characters that need to be case folded and then normalized.

2.5 Character Escapes and Includes

Most document formats or protocols provide an escaping mechanism to permit the inclusion of characters that are otherwise difficult to input, process, or encode. These escaping mechanisms provide an additional equivalent means of representing characters inside a given resource. They also allow for the encoding of Unicode characters not represented in the character encoding scheme used by the document.

For example, € U+20AC EURO SIGN can also be encoded in HTML as the hexadecimal entity € or as the decimal entity €. In a JavaScript or JSON file, it can appear as \u20ac or as \u{20AC} while in a CSS stylesheet it can appear as \20ac. All of these representations encode the same literal character value: €.

Character escapes are normally interpreted before a document is processed and strings within the format or protocol are matched. Returning to an example we used above:

You would expect that text to display like the following: Hello world!

In order for this to work, the user-agent (browser) had to match two strings representing the class name héllo, even though the CSS and HTML each used a different escaping mechanism. The above fragment demonstrates one way that text can vary and still be considered "the same" according to a specification: the class name h\e9llo matched the class name in the HTML mark-up héllo (and would also match the literal value héllo using the code point é [U+00E9 LATIN SMALL LETTER E WITH ACUTE]).

Formal languages and document formats often offer facilities for including a piece of text from one resource inside another. An include is a mechanism for inserting content into the body of a resource. Include mechanisms import content into a resource at processing time. This affects the structure of the document and potentially matching against the vocabulary of the document. Examples of includes are entity references in XML, the XInclude [XInclude] specification, and @import rules in CSS.

An include is said to be include normalized if it does not begin with a combining mark (either in the form of a character escape or as a character literal in the included resource).

2.6 Invisible Unicode Characters

Unicode provides a number of special-purpose characters that help document authors control the appearance or performance of text. Because many of these characters are invisible or do not have keyboard equivalents, users are not always aware of their presence or absence. As a result, these characters can interfere with string matching when they are part of the encoded character sequence but the expected matching text does not also include them. Some examples of these characters include:

The Unicode control characters U+200D Zero Width Joiner (also known as ZWJ) and U+200C Zero Width Non-Joiner (also known as ZWNJ). While these characters can be used to control ligature formation—either preventing the formation of undesirable ligatures or encouraging the formation of desirable ones—their primary use is to control the joining and shape selection in complex scripts such as the Arabic or various of the Indic scripts. Some Indic scripts use the ZWJ and ZWNJ characters to allow authors to control the shape that certain conjuncts take. See the discussion in Chapter 12 of [Unicode].

The Zero Width Non-Joiner is used in Persian to prevent certain "normal" Arabic script joining. In these cases, the presence or absence of the character does affect the meaning. For example, the word تنها ("alone") and the word تن‌ها  ("bodies" or "corpuses") are encoded as "U+062A U+0646 U+0647 U+0627" and "U+062A U+0646 U+200C U+0647 U+0627" respectively, the only difference being the ZWNJ in the latter word.

The ZWJ character is also used in forming certain emoji sequences, which is discussed in more detail below.

Variation selectors (U+FE00 through U+FE0F) are characters used to select an alternate appearance or glyph (see Character Model: Fundamentals [CHARMOD]). For example, they are used to select between black-and-white and color emoji. These are also used in predefined ideographic variation sequences (IVS). Many examples are given in the "Standardized Variants" portion of the Unicode Character Database (UCD).

A few scripts also provide a way to encode visual variation selection: a prominent example of this are the Mongolian script's free variation selectors (U+180B through U+180D).

The character U+034F Combining Grapheme Joiner, whose name is misleading (as it does not join graphemes), is used to separate characters that might otherwise be considered a grapheme for the purposes of sorting or to provide a means of maintaing certain textual distinctions when applying Unicode normalization to text.

Whitespace variations can also affect the interpretation and matching of text. For example, the various non-breaking space characters, such as NBSP, NNBSP, etc.

U+200B Zero Width Space is a character used to indicate word boundaries in text where spaces do not otherwise appear. For example, it might be used in a Thai language document to assist with word-breaking.

The U+00AD Soft Hyphen can be used in text to indicate a potential or preferred hyphenation position. It only becomes visible when the text is reflowed to wrap at that position.

The U+2060 WORD JOINER, sometimes called WJ, is a zero-width non-breaking space character. Its purpose is to prevent line breaks between two characters. Except for purposes of line-breaking, it should be ignored. It serves as a replacement for the character U+FEFF ZERO WIDTH NO-BREAK SPACE because U+FEFF is more commonly known as the "Byte Order Mark" (BOM). A byte order mark is used at the start of some plain text files to signal that the file is in a Unicode character encoding.

Finally, most scripts, when written horizontally, proceed from left-to-right. However, some scripts, such as Arabic and Hebrew, are written predominently from right-to-left. Texts can be written in a mix of these scripts or include character sequences, such as numbers or quotes in another script, that run in the opposite direction to other parts of the text. This intermixing of text direction is called bidirectional text or bidi for short. The Unicode Bidirectional Algorithm [UAX9] describes how such mixed-direction text is processed for display. For most text, the directional handling can be derived from the text itself. However, there are many cases in which the algorithm needs additional information in order to present text correctly. For more examples, see [html-bidi].

One of the ways that Unicode defines to address the ambiguity of text direction are a set of invisible control characters to mark the start and end of directional runs. While bidirectional controls can have an affect on the appearance of the text (since they help the Unicode Bidirectional Algorithm with the presentation of text), they might have no effect on the text if the text would naturally have fallen into bidirectional runs without the controls. Because these controls are, like the characters mentioned above, invisible, they can have an unintentional effect on matching.

In almost all of these cases, users may not be aware of or cannot be sure if a given document or text string has included or omitted one of these characters. Because text matching depends on matching the underlying codepoints, variation in the encoding of the text due to these markers can cause matches that ought to succeed to mysteriously fail (from the point of view of the user).

2.7 Emoji Sequences

A newer feature of Unicode are the emoji characters. In [UTR51], Unicode describes these as:

Emoji are pictographs (pictorial symbols) that are typically presented in a colorful cartoon form and used inline in text. They represent things such as faces, weather, vehicles and buildings, food and drink, animals and plants, or icons that represent emotions, feelings, or activities.

Emoji can be used with a variety of emoji modifiers, including U+200D ZERO WIDTH JOINER or ZWJ, to form more complex emoji.

For example, the emoji (👪 [U+1F46A FAMILY]) can also be formed by using ZWJ between emoji characters in the sequence U+1F468 U+200D U+1F469 U+200D U+1F466. Altering or adding other emoji characters can alter the composition of the family. For example the sequence 👨‍👩‍👧‍👧 U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F467 results in a composed emoji character for a "family: man, woman, girl, girl" on systems that support this kind of composition. Many common emoji can only be formed using ZWJ sequences. For more information, see [UTR51].

Emoji characters can be followed by emoji modifier characters. These modifiers allow for the selection of skin tones for emoji that represent people. These characters are normally invisible modifiers that follow the base emoji that they modify. For example: 👨 👨🏻 👨🏼 👨🏽 👨🏾 👨🏿

An emoji character can also be followed by a variation selector to indicate text (black and white, indicated by U+FF0E Variation Selector 15) or color (indicated by U+FF0F Variation Selector 16) presentation of the base emoji.

Still another wrinkle in the use of emoji are flags. National flags can be composed using country codes derived from the [BCP47] registry, such as the sequence 🇿 [U+1F1FF REGIONAL INDICATOR SYMBOL LETTER Z] 🇲 [U+1F1F2 REGIONAL INDICATOR SYMBOL LETTER M], which is the country code (ZM) for the country Zambia: 🇿🇲. Other regional or special purpose flags can be composed using a flag emoji with various symbols or with regional indicator codes terminating in a cancel tag. For example, the flag of Scotland (🏴󠁧󠁢󠁳󠁣󠁴󠁿) can be composed like this:

• 🏴 [U+1F3F4 WAVING BLACK FLAG]
• 󠁧 [U+E0067 TAG LATIN SMALL LETTER G]
• 󠁢 [U+E0062 TAG LATIN SMALL LETTER B]
• 󠁳 [U+E0073 TAG LATIN SMALL LETTER S]
• 󠁣 [U+E0063 TAG LATIN SMALL LETTER C]
• 󠁴 [U+E0074 TAG LATIN SMALL LETTER T]
• 󠁿 [U+E007F CANCEL TAG]

Each of these mechanisms can be used together, so quite complex sequences of characters can be used to form a single emoji grapheme or image. Even very similar emoji sequences might not use the same exact encoded sequence. In most cases the modifiers and combinations mentioned above are generated by the end-user's keyboard (where they are presented as a single emoji "character"). This diversity of encoding options is partially addressed to the extent that different vendors use only (and exactly) the sequences that are "recommended for interchange" by Unicode. This helps vendors ensure that fonts and keyboards are prepared to give users the options they expect. Still, users generally won't be aware of the underlying encoding complexity and generative mechanisms are not limited to those that are recommended. Emoji sequences are evolving rapidly, so there could be additional developments to either help or hinder matching of emoji in the near future. Unicode normalization does not reorder these sequences or insert or remove any of the modifiers. Users and implementers are therefore cautioned that users who employ emoji characters in namespaces and other matching contexts can easily encounter unexpected "character" mismatches due to variations in encoding.

2.8 Legacy Character Encodings

Resources can use different character encoding schemes, including legacy character encodings, to serialize document formats on the Web. Each character encoding scheme uses different byte values and sequences to represent a given subset of the Universal Character Set.

For example, € [U+20AC EURO SIGN]) is encoded as the byte sequence 0xE2.82.AC in the UTF-8 character encoding. This same character is encoded as the byte sequence 0x80 in the legacy character encoding windows-1252. (Other legacy character encodings may not provide any byte sequence to encode the character.)

Specifications mainly address these resulting variations by considering each document to be a sequence of Unicode characters after converting from the document's character encoding (be it a legacy character encoding or a Unicode encoding such as UTF-8) and then unescaping any character escapes before proceeding to process the document.

One additional consideration in converting to Unicode is the existence of legacy character encodings of bidirectional scripts (such as Hebrew and Arabic) that use a visual storage order. That is, unlike Unicode and other modern encodings, the characters are stored in memory in the order that they are printed on the screen from left-to-right (as with a line printer). When converting these encodings to Unicode or when comparing text in these encodings, care must be taken to place both the source and target text into logical order. For more information, see Section 3.3.1 of [CHARMOD]

2.9 Other Types of Equivalence

3.1 Choosing a Matching Algorithm

The basic decision for choosing a matching algorithm to use for a given specification is the level of text normalization (which includes both case sensitivity and Unicode normalization) to apply to the strings being matched. Historically on the Web most specifications have opted for case-sensitive matching without Unicode normalization and this is the RECOMMENDED form of matching for all new specifications. However, there are cases where case-insensitivity and normalization are useful.

A specificiation can choose to be case-insensitive if the benefit to users of the format or protocol in question outweighs the cost and complexity of the implementation. Because both case folding and normalization can affect the values being compared, including the presentation and, in some cases, the meaning of the text and because these operations are relatively expensive, choosing case-insensitivity is generally discouraged.

A special case of case-insensitivity are vocabularies limited to the ASCII/Basic Latin range (that is, code points U+0000 to U+007F). These specifications can choose to be case-insensitive only over that range of characters. This greatly simplifies the implementation of matching. However, this form of matching is not appropriate for specifications that allow a larger range of Unicode in identifiers or syntax, since it is difficult for users to understand the behavior of the matching and it is a disadvantage to users of non-ASCII scripts and languages. That is, users find it to be weird and unpredictable when green matches GREEN but grüß doesn't match GRÜẞ or possibly GRÜSS (but instead matches GRüß).

4.1 Regular Expressions

[S] Specifications that define a regular expression syntax MUST provide at least Basic Unicode Level 1 support per [UTS18] and SHOULD provide Extended or Tailored (Levels 2 and 3) support.

Regular expression syntaxes are sometimes useful in defining a format or protocol, since they allow users to specify values that are only partially known or which can vary in predictable ways. As seen in the various sections of this document, there is variation in the different ways that characters can be encoded in Unicode and this potentially interferes with how strings are specified or matched in expressions. For example, counting characters might need to depend on grapheme boundaries rather than the number of Unicode code points used; caseless matching might need to consider variations in case folding; or the Unicode normalization of the expression or text being processed might need to be considered.

Unicode Regular Expressions Level 1 support includes the ability to specify Unicode code points in regular expressions, including via the use of escapes, and to access Unicode character properties as well as certain kinds of boundaries common to most regular expression syntaxes.

Level 2 extends this with a number of important capabilities, notably the ability to select text on certain kinds of grapheme cluster boundary and support for case conversion (two topics mentioned extensively above). Level 3 provides for locale [LTLI] based tailoring of regular expressions, which are less useful in formal languages but can be useful in processing natural language content.