<!-- X-URL:
http://www.unicode.org/Public/UNIDATA/UnicodeData.html -->
<!-- Date: Sun, 04 Aug 2002 09:12:51 GMT -->
<!-- Last-Modified: Wed, 27 Mar 2002 22:55:34 GMT -->
<BASE HREF="
http://www.unicode.org/Public/UNIDATA/UnicodeData.html">
<html>
<head>
<meta name="GENERATOR" content="Microsoft FrontPage 4.0">
<meta name="ProgId" content="FrontPage.Editor.Document">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>UnicodeData File Format</title>
<link rel="stylesheet" type="text/css" href="
http://www.unicode.org/reports/reports.css">
</head>
<body bgcolor="#ffffff">
<table class="header" width="100%">
<tr>
<td class="icon"><a href="
http://www.unicode.org"><img align="middle" alt="[Unicode]" border="0" src="
http://www.unicode.org/webscripts/logo60s2.gif" width="34" height="33"></a> <a class="bar" href="UnicodeCharacterDatabase.html">Unicode
Character Database</a></td>
</tr>
<tr>
<td class="gray"> </td>
</tr>
</table>
<blockquote>
<h1>Unicode Data File Format</h1>
<table class="wide" border="1">
<tr>
<td valign="TOP" width="144">Revision</td>
<td valign="TOP">3.2.0</td>
</tr>
<tr>
<td valign="TOP" width="144">Authors</td>
<td valign="TOP">Mark Davis and Ken Whistler</td>
</tr>
<tr>
<td valign="TOP" width="144">Date</td>
<td valign="TOP">2002-03-22</td>
</tr>
<tr>
<td valign="TOP" width="144">This Version</td>
<td valign="TOP"><a href="
http://www.unicode.org/Public/3.2-Update/UnicodeData-3.2.0.html">
http://www.unicode.org/Public/3.2-Update/UnicodeData-3.2.0.html</a></td>
</tr>
<tr>
<td valign="TOP" width="144">Previous Version</td>
<td valign="TOP"><a href="
http://www.unicode.org/Public/3.1-Update/UnicodeData-3.1.0.html">
http://www.unicode.org/Public/3.1-Update/UnicodeData-3.1.0.html</a></td>
</tr>
<tr>
<td valign="TOP" width="144">Latest Version</td>
<td valign="TOP"><a href="
http://www.unicode.org/Public/UNIDATA/UnicodeData.html">
http://www.unicode.org/Public/UNIDATA/UnicodeData.html</a></td>
</tr>
</table>
<h3><br>
S<i>ummary</i></h3>
<blockquote>
<p><i>This document describes the format and content of the UnicodeData.txt
file in the Unicode Character Database (UCD).</i></p>
</blockquote>
<h3><i>Status</i></h3>
<blockquote>
<p><i>The file and the files described herein are part of the Unicode
Character Database and governed by the <a href="#UCD_Terms">UCD Terms
of Use</a> given below.</i></p>
<p><i>For general information on file formats and table formats, and the
implications of normative vs informative properties, see
UnicodeCharacterDatabase.html.</i></p>
<p><i><b>Warning: </b>the information in this file does not completely
describe the use and interpretation of Unicode character properties and
behavior. It must be used in conjunction with the data in the other files in
the Unicode Character Database, and relies on the notation and definitions
supplied in <a href="
http://www.unicode.org/standard/standard.html">The
Unicode Standard</a>. All chapter references are to Version 3.2.0 of the
standard unless otherwise indicated.</i></p>
</blockquote>
<h2>Contents</h2>
<ul>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#Field Formats">Field Formats</a>
<ul>
<li><a href="#General Category">General Category</a></li>
<li><a href="#Bidirectional Category">Bidirectional Category</a></li>
<li><a href="#Character Decomposition">Character Decomposition Mapping</a></li>
<li><a href="#Canonical Combining Classes">Canonical Combining Classes</a></li>
<li><a href="#Decompositions and Normalization">Decompositions and
Normalization</a></li>
<li><a href="#Case Mappings">Case Mappings</a></li>
</ul>
</li>
<li><a href="#Property Invariants">Property Invariants</a></li>
<li><a href="#Modification History">Modification History</a></li>
</ul>
<h2><a name="Introduction">Introduction</a></h2>
<p>This document describes the format of the UnicodeData.txt file, which is
one of the most important files in the Unicode Character Database.
<h2><a name="Field Formats"></a>Field Formats</h2>
<p>Each line represents the data for one encoded character in the Unicode
Standard. (For information on the file format, see UCD File Format in
UnicodeCharacterDatabase.html).
<p>Every encoded character has a data entry, with the exception of certain
special ranges, as detailed below.
<ul>
<li>These ranges represented only by their start and end characters, since
the properties in the file are uniform, except for code points (which are
all sequential and assigned) and names.</li>
<li>The names of CJK ideograph characters and the names and decompositions
of Hangul syllable characters are algorithmically derivable. (See the
Unicode Standard and Unicode Standard Annex #15 for more information).</li>
<li>Surrogate code points and private use characters have no names.</li>
<li>The supplementary Private Use characters (U+F0000 .. U+FFFFD, U+100000
.. U+10FFFD) are listed as distinct ranges.</li>
</ul>
<p>The exact ranges represented by start and end characters are:
<ul>
<li>CJK Ideographs Extension A (U+3400 .. U+4DB5)</li>
<li>CJK Ideographs (U+4E00 .. U+9FA5)</li>
<li>Hangul Syllables (U+AC00 .. U+D7A3)</li>
<li>Non-Private Use High Surrogates (U+D800 .. U+DB7F)</li>
<li>Private Use High Surrogates (U+DB80 .. U+DBFF)</li>
<li>Low Surrogates (U+DC00 .. U+DFFF)</li>
<li>The Private Use Area (U+E000 .. U+F8FF)</li>
<li>CJK Ideographs Extension B (U+20000 .. U+2A6D6)</li>
<li>Plane 15 Private Use Area (U+F0000 .. U+FFFFD)</li>
<li>Plane 16 Private Use Area (U+100000 .. U+10FFFD)</li>
</ul>
<p>The following table describes the format and meaning of each field in a
data entry in the UnicodeData file.</p>
<table>
<tr>
<th valign="top" align="LEFT">
<p align="LEFT">Field</th>
<th valign="top" align="LEFT">
<p align="LEFT">Name</th>
<th valign="top" align="LEFT">
<p align="LEFT">N/I</th>
<th valign="top" align="LEFT">
<p align="LEFT">Explanation</th>
</tr>
<tr>
<th valign="top">0</th>
<td valign="top">Code point</td>
<td valign="top" align="center">N</td>
<td valign="top">Code point.</td>
</tr>
<tr>
<th valign="top">1</th>
<td valign="top">Character name</td>
<td valign="top" align="center">N</td>
<td valign="top">These names match exactly the names published in the code
charts of the Unicode Standard.</td>
</tr>
<tr>
<th valign="top">2</th>
<td valign="top"><a href="#General Category">General Category</a></td>
<td valign="top" align="center">N</td>
<td valign="top">This is a useful breakdown into various "character
types" which can be used as a default categorization in
implementations. See below for a brief explanation.</td>
</tr>
<tr>
<th valign="top">3</th>
<td valign="top"><a href="#Canonical Combining Classes">Canonical
Combining Classes</a></td>
<td valign="top" align="center">N</td>
<td valign="top">The classes used for the Canonical Ordering Algorithm in
the Unicode Standard. These classes are also printed in Chapter 4 of the
Unicode Standard.</td>
</tr>
<tr>
<th valign="top">4</th>
<td valign="top"><a href="#Bidirectional Category">Bidirectional Category</a></td>
<td valign="top" align="center">N</td>
<td valign="top">See the list below for an explanation of the
abbreviations used in this field. These are the categories required by
the Bidirectional Behavior Algorithm in the Unicode Standard. These
categories are summarized in Chapter 3 of the Unicode Standard.</td>
</tr>
<tr>
<th valign="top">5</th>
<td valign="top"><a href="#Character Decomposition">Character
Decomposition Mapping</a></td>
<td valign="top" align="center">N</td>
<td valign="top">In the Unicode Standard, not all of the mappings are full
(maximal) decompositions. Recursive application of look-up for
decompositions will, in all cases, lead to a maximal decomposition. The
decomposition mappings match exactly the decomposition mappings
published with the character names in the Unicode Standard.</td>
</tr>
<tr>
<th valign="top">6</th>
<td valign="top">Decimal digit value</td>
<td valign="top" align="center">N</td>
<td valign="top">This is a numeric field. If the character has the decimal
digit property, as specified in Chapter 4 of the Unicode Standard, the
value of that digit is represented with an integer value in this field</td>
</tr>
<tr>
<th valign="top">7</th>
<td valign="top">Digit value</td>
<td valign="top" align="center">N</td>
<td valign="top">This is a numeric field. If the character represents a
digit, not necessarily a decimal digit, the value is here. This covers
digits which do not form decimal radix forms, such as the compatibility
superscript digits</td>
</tr>
<tr>
<th valign="top">8</th>
<td valign="top">Numeric value</td>
<td valign="top" align="center">N</td>
<td valign="top">This is a numeric field. If the character has the numeric
property, as specified in Chapter 4 of the Unicode Standard, the value
of that character is represented with an integer or rational number in
this field. This includes fractions as, e.g., "1/5" for U+2155
VULGAR FRACTION ONE FIFTH Also included are numerical values for
compatibility characters such as circled numbers.</td>
</tr>
<tr>
<th valign="top">9</th>
<td valign="top">Mirrored</td>
<td valign="top" align="center">N</td>
<td valign="top">If the character has been identified as a
"mirrored" character in bidirectional text, this field has the
value "Y"; otherwise "N". The list of mirrored
characters is also printed in Chapter 4 of the Unicode Standard.</td>
</tr>
<tr>
<th valign="top">10</th>
<td valign="top">Unicode 1.0 Name</td>
<td valign="top" align="center">I</td>
<td valign="top">This is the old name as published in Unicode 1.0. This
name is only provided when it is significantly different from the
current name for the character. The value of field 10 for control
characters does not always match the Unicode 1.0 names. Instead, field
10 contains ISO 6429 names for control functions, for printing in the
code charts.</td>
</tr>
<tr>
<th valign="top">11</th>
<td valign="top">10646 comment field</td>
<td valign="top" align="center">I</td>
<td valign="top">This is the ISO 10646 comment field. It appears in
parentheses in the 10646 names list, or contains an asterisk to mark an
Annex P note.</td>
</tr>
<tr>
<th valign="top">12</th>
<td valign="top"><a href="#Case Mappings">Uppercase Mapping</a></td>
<td valign="top" align="center">N</td>
<td valign="top">Upper case equivalent mapping. If a character is part of
an alphabet with case distinctions, and has a simple upper case
equivalent, then the upper case equivalent is in this field. See the
explanation below on case distinctions. These mappings are always
one-to-one, not one-to-many or many-to-one.
<p><i><b>Note: </b>This field is omitted if the uppercase is the same as
field 0.</i> <i>For full case mappings, see UAX #21 Case Mappings and
SpecialCasing.txt.</i></td>
</tr>
<tr>
<th valign="top">13</th>
<td valign="top"><a href="#Case Mappings">Lowercase Mapping</a></td>
<td valign="top" align="center">N</td>
<td valign="top">Similar to Uppercase mapping
<p><i><b>Note: </b>This field is omitted if the lowercase is the same as
field 0.</i> <i>For full case mappings, see UAX #21 Case Mappings and
SpecialCasing.txt.</i></p>
</td>
</tr>
<tr>
<th valign="top">14</th>
<td valign="top"><a href="#Case Mappings">Titlecase Mapping</a></td>
<td valign="top" align="center">N</td>
<td valign="top">Similar to Uppercase mapping.
<p><i><b>Note: </b>This field is omitted if the titlecase is the same as
field 12.</i> <i>For full case mappings, see UAX #21 Case Mappings and
SpecialCasing.txt.</i></p>
</td>
</tr>
</table>
<h3><a name="General Category"></a>General Category</h3>
<p>The values in this field are abbreviations for the following values. For
more information, see the Unicode Standard.</p>
<blockquote>
<p><b>Note:</b> the standard does not assign information to control
characters (except for certain cases in the Bidirectional Algorithm).
Implementations will generally also assign categories to certain control
characters, notably CR and LF, according to platform conventions. See UAX
#13: Unicode Newline Guidelines for more information.</p>
</blockquote>
<table>
<tr>
<th>
<p align="LEFT">Abbr.</th>
<th>
<p align="LEFT">Description</th>
</tr>
<tr>
<td align="CENTER">Lu</td>
<td>Letter, Uppercase</td>
</tr>
<tr>
<td align="CENTER">Ll</td>
<td>Letter, Lowercase</td>
</tr>
<tr>
<td align="CENTER">Lt</td>
<td>Letter, Titlecase</td>
</tr>
<tr>
<td align="CENTER">Lm</td>
<td>Letter, Modifier</td>
</tr>
<tr>
<td align="CENTER">Lo</td>
<td>Letter, Other</td>
</tr>
<tr>
<td align="CENTER">Mn</td>
<td>Mark, Non-Spacing</td>
</tr>
<tr>
<td align="CENTER">Mc</td>
<td>Mark, Spacing Combining</td>
</tr>
<tr>
<td align="CENTER">Me</td>
<td>Mark, Enclosing</td>
</tr>
<tr>
<td align="CENTER">Nd</td>
<td>Number, Decimal Digit</td>
</tr>
<tr>
<td align="CENTER">Nl</td>
<td>Number, Letter</td>
</tr>
<tr>
<td align="CENTER">No</td>
<td>Number, Other</td>
</tr>
<tr>
<td align="CENTER">Pc</td>
<td>Punctuation, Connector</td>
</tr>
<tr>
<td align="CENTER">Pd</td>
<td>Punctuation, Dash</td>
</tr>
<tr>
<td align="CENTER">Ps</td>
<td>Punctuation, Open</td>
</tr>
<tr>
<td align="CENTER">Pe</td>
<td>Punctuation, Close</td>
</tr>
<tr>
<td align="CENTER">Pi</td>
<td>Punctuation, Initial quote (may behave like Ps or Pe depending on
usage)</td>
</tr>
<tr>
<td align="CENTER">Pf</td>
<td>Punctuation, Final quote (may behave like Ps or Pe depending on usage)</td>
</tr>
<tr>
<td align="CENTER">Po</td>
<td>Punctuation, Other</td>
</tr>
<tr>
<td align="CENTER">Sm</td>
<td>Symbol, Math</td>
</tr>
<tr>
<td align="CENTER">Sc</td>
<td>Symbol, Currency</td>
</tr>
<tr>
<td align="CENTER">Sk</td>
<td>Symbol, Modifier</td>
</tr>
<tr>
<td align="CENTER">So</td>
<td>Symbol, Other</td>
</tr>
<tr>
<td align="CENTER">Zs</td>
<td>Separator, Space</td>
</tr>
<tr>
<td align="CENTER">Zl</td>
<td>Separator, Line</td>
</tr>
<tr>
<td align="CENTER">Zp</td>
<td>Separator, Paragraph</td>
</tr>
<tr>
<td align="CENTER">Cc</td>
<td>Other, Control</td>
</tr>
<tr>
<td align="CENTER">Cf</td>
<td>Other, Format</td>
</tr>
<tr>
<td align="CENTER">Cs</td>
<td>Other, Surrogate</td>
</tr>
<tr>
<td align="CENTER">Co</td>
<td>Other, Private Use</td>
</tr>
<tr>
<td align="CENTER">Cn</td>
<td>Other, Not Assigned (no characters in the file have this property)</td>
</tr>
</table>
<blockquote>
<p><b>Note:</b> The term "L&" is sometimes used to stand for
Uppercase, Lowercase or Titlecase letters (Lu, Ll, or Lt) in comments. This
is the same as the LC value in PropertyValueAliases.txt.</p>
</blockquote>
<h3><a name="Bidirectional Category"></a>Bidirectional Category</h3>
<p>Please refer to Chapter 3 for an explanation of the algorithm for
Bidirectional Behavior and an explanation of the significance of these
categories. An up-to-date version can be found on UAX #9: The Bidirectional
Algorithm.</p>
<table>
<tr>
<th valign="TOP" align="LEFT">
<p align="LEFT">Type</th>
<th valign="TOP" align="LEFT">
<p align="LEFT">Description</th>
</tr>
<tr>
<td valign="TOP"><b>L</b></td>
<td valign="TOP">Left-to-Right</td>
</tr>
<tr>
<td valign="TOP"><b>LRE</b></td>
<td valign="TOP">Left-to-Right Embedding</td>
</tr>
<tr>
<td valign="TOP"><b>LRO</b></td>
<td valign="TOP">Left-to-Right Override</td>
</tr>
<tr>
<td valign="TOP"><b>R</b></td>
<td valign="TOP">Right-to-Left</td>
</tr>
<tr>
<td valign="TOP"><b>AL</b></td>
<td valign="TOP">Right-to-Left Arabic</td>
</tr>
<tr>
<td valign="TOP"><b>RLE</b></td>
<td valign="TOP">Right-to-Left Embedding</td>
</tr>
<tr>
<td valign="TOP"><b>RLO</b></td>
<td valign="TOP">Right-to-Left Override</td>
</tr>
<tr>
<td valign="TOP"><b>PDF</b></td>
<td valign="TOP">Pop Directional Format</td>
</tr>
<tr>
<td valign="TOP"><b>EN</b></td>
<td valign="TOP">European Number</td>
</tr>
<tr>
<td valign="TOP"><b>ES</b></td>
<td valign="TOP">European Number Separator</td>
</tr>
<tr>
<td valign="TOP"><b>ET</b></td>
<td valign="TOP">European Number Terminator</td>
</tr>
<tr>
<td valign="TOP"><b>AN</b></td>
<td valign="TOP">Arabic Number</td>
</tr>
<tr>
<td valign="TOP"><b>CS</b></td>
<td valign="TOP">Common Number Separator</td>
</tr>
<tr>
<td valign="TOP"><b>NSM</b></td>
<td valign="TOP">Non-Spacing Mark</td>
</tr>
<tr>
<td valign="TOP"><b>BN</b></td>
<td valign="TOP">Boundary Neutral</td>
</tr>
<tr>
<td valign="TOP"><b>B</b></td>
<td valign="TOP">Paragraph Separator</td>
</tr>
<tr>
<td valign="TOP"><b>S</b></td>
<td valign="TOP">Segment Separator</td>
</tr>
<tr>
<td valign="TOP"><b>WS</b></td>
<td valign="TOP">Whitespace</td>
</tr>
<tr>
<td valign="TOP"><b>ON</b></td>
<td valign="TOP">Other Neutrals</td>
</tr>
</table>
<h3><a name="Character Decomposition"></a>Character Decomposition Mapping</h3>
<p>The tags supplied with certain decomposition mappings generally indicate
formatting information. Where no such tag is given, the mapping is designated
as canonical. Conversely, the presence of a formatting tag also indicates that
the mapping is a compatibility mapping and not a canonical mapping. In the
absence of other formatting information in a compatibility mapping, the tag is
used to distinguish it from canonical mappings.</p>
<p>In some instances a canonical mapping or a compatibility mapping may
consist of a single character. For a canonical mapping, this indicates that
the character is a canonical equivalent of another single character. For a
compatibility mapping, this indicates that the character is a compatibility
equivalent of another single character. The compatibility formatting tags used
are:</p>
<table>
<tr>
<th>Tag</th>
<th>
<p align="LEFT">Description</th>
</tr>
<tr>
<td align="CENTER"><font> </td>
<td>A font variant (e.g. a blackletter form).</td>
</tr>
<tr>
<td align="CENTER"><noBreak> </td>
<td>A no-break version of a space or hyphen.</td>
</tr>
<tr>
<td align="CENTER"><initial> </td>
<td>An initial presentation form (Arabic).</td>
</tr>
<tr>
<td align="CENTER"><medial> </td>
<td>A medial presentation form (Arabic).</td>
</tr>
<tr>
<td align="CENTER"><final> </td>
<td>A final presentation form (Arabic).</td>
</tr>
<tr>
<td align="CENTER"><isolated> </td>
<td>An isolated presentation form (Arabic).</td>
</tr>
<tr>
<td align="CENTER"><circle> </td>
<td>An encircled form.</td>
</tr>
<tr>
<td align="CENTER"><super> </td>
<td>A superscript form.</td>
</tr>
<tr>
<td align="CENTER"><sub> </td>
<td>A subscript form.</td>
</tr>
<tr>
<td align="CENTER"><vertical> </td>
<td>A vertical layout presentation form.</td>
</tr>
<tr>
<td align="CENTER"><wide> </td>
<td>A wide (or zenkaku) compatibility character.</td>
</tr>
<tr>
<td align="CENTER"><narrow> </td>
<td>A narrow (or hankaku) compatibility character.</td>
</tr>
<tr>
<td align="CENTER"><small> </td>
<td>A small variant form (CNS compatibility).</td>
</tr>
<tr>
<td align="CENTER"><square> </td>
<td>A CJK squared font variant.</td>
</tr>
<tr>
<td align="CENTER"><fraction> </td>
<td>A vulgar fraction form.</td>
</tr>
<tr>
<td align="CENTER"><compat> </td>
<td>Otherwise unspecified compatibility character.</td>
</tr>
</table>
<p><b>Reminder: </b>There is a difference between decomposition and
decomposition mapping. The decomposition mappings are defined in the
UnicodeData, while the decomposition (also termed "full
decomposition") is defined in Chapter 3 to use those mappings <i>recursively.</i>
<ul>
<li>The canonical decomposition is formed by recursively applying the
canonical mappings, then applying the canonical reordering algorithm.</li>
<li>The compatibility decomposition is formed by recursively applying the
canonical <em>and</em> compatibility mappings, then applying the canonical
reordering algorithm.</li>
</ul>
<h3><a name="Canonical Combining Classes"></a>Canonical Combining Classes</h3>
<table>
<tr>
<th>
<p align="LEFT">Value</th>
<th>
<p align="LEFT">Description</th>
</tr>
<tr>
<td align="RIGHT">0:</td>
<td>Spacing, split, enclosing, reordrant, and Tibetan subjoined</td>
</tr>
<tr>
<td align="RIGHT">1:</td>
<td>Overlays and interior</td>
</tr>
<tr>
<td align="RIGHT">7:</td>
<td>Nuktas</td>
</tr>
<tr>
<td align="RIGHT">8:</td>
<td>Hiragana/Katakana voicing marks</td>
</tr>
<tr>
<td align="RIGHT">9:</td>
<td>Viramas</td>
</tr>
<tr>
<td align="RIGHT">10:</td>
<td>Start of fixed position classes</td>
</tr>
<tr>
<td align="RIGHT">199:</td>
<td>End of fixed position classes</td>
</tr>
<tr>
<td align="RIGHT">200:</td>
<td>Below left attached</td>
</tr>
<tr>
<td align="RIGHT">202:</td>
<td>Below attached</td>
</tr>
<tr>
<td align="RIGHT">204:</td>
<td>Below right attached</td>
</tr>
<tr>
<td align="RIGHT">208:</td>
<td>Left attached (reordrant around single base character)</td>
</tr>
<tr>
<td align="RIGHT">210:</td>
<td>Right attached</td>
</tr>
<tr>
<td align="RIGHT">212:</td>
<td>Above left attached</td>
</tr>
<tr>
<td align="RIGHT">214:</td>
<td>Above attached</td>
</tr>
<tr>
<td align="RIGHT">216:</td>
<td>Above right attached</td>
</tr>
<tr>
<td align="RIGHT">218:</td>
<td>Below left</td>
</tr>
<tr>
<td align="RIGHT">220:</td>
<td>Below</td>
</tr>
<tr>
<td align="RIGHT">222:</td>
<td>Below right</td>
</tr>
<tr>
<td align="RIGHT">224:</td>
<td>Left (reordrant around single base character)</td>
</tr>
<tr>
<td align="RIGHT">226:</td>
<td>Right</td>
</tr>
<tr>
<td align="RIGHT">228:</td>
<td>Above left</td>
</tr>
<tr>
<td align="RIGHT">230:</td>
<td>Above</td>
</tr>
<tr>
<td align="RIGHT">232:</td>
<td>Above right</td>
</tr>
<tr>
<td align="RIGHT">233:</td>
<td>Double below</td>
</tr>
<tr>
<td align="RIGHT">234:</td>
<td>Double above</td>
</tr>
<tr>
<td align="RIGHT">240:</td>
<td>Below (iota subscript)</td>
</tr>
</table>
<blockquote>
<p><strong>Note: </strong>some of the combining classes in this list do not
currently have members but are specified here for completeness.</p>
</blockquote>
<h3><a name="Decompositions and Normalization"></a>Decompositions and
Normalization</h3>
<p>Decomposition is specified in Chapter 3. <i>UAX #15: Unicode Normalization
Forms</i> specifies the interaction between decomposition and normalization.
That report specifies how the decompositions defined in UnicodeData.txt are
used to derive normalized forms of Unicode text.</p>
<p>Note that as of the 2.1.9 update of the Unicode Character Database, the
decompositions in the UnicodeData.txt file can be used to recursively derive
the full decomposition in canonical order, without the need to separately
apply canonical reordering. However, canonical reordering of combining
character sequences <b><i>must</i></b> still be applied in decomposition when
normalizing source text which contains any combining marks.</p>
<h3><a name="Case Mappings"></a>Case Mappings</h3>
<p>There are a number of complications to case mappings that occur once the
repertoire of characters is expanded beyond ASCII. For more information, see
UAX #21: Case Mappings.</p>
<p>For compatibility with existing parsers, UnicodeData.txt only contains case
mappings for characters where they are one-to-one mappings; it also omits
information about context-sensitive case mappings. Information about these
special cases can be found in a separate data file, SpecialCasing.txt.</p>
<h2><a name="Property Invariants"></a>Property Invariants</h2>
<p>Values in UnicodeData.txt are subject to correction as errors are found;
however, some characteristics of the categories themselves can be considered
invariants. Applications may wish to take these invariants into account when
choosing how to implement character properties. For more information, see <a href="
http://www.unicode.org/unicode/standard/policies.html">Unicode
Policies</a>.</p>
<p>The following is a partial list of known invariants for the Unicode
Character Database.</p>
<h4>Database Fields</h4>
<ul>
<li>The number of fields in UnicodeData.txt is fixed.</li>
<li>The order of the fields is also fixed.
<ul>
<li>Any additional information about character properties to be added in
the future will appear in separate data tables, rather than being
added on to the existing table or by subdivision or reinterpretation
of existing fields.</li>
</ul>
</li>
</ul>
<h4>General Category</h4>
<ul>
<li>There will never be more than 32 General Category values.
<ul>
<li>It is very unlikely that the Unicode Technical Committee will
subdivide the General Category partition any further, since that can
cause implementations to misbehave. Because the General Category is
limited to 32 values, 5 bits can be used to represent the information,
and a 32-bit integer can be used as a bitmask to represent arbitrary
sets of categories.</li>
</ul>
</li>
</ul>
<h4>Combining Classes</h4>
<ul>
<li>Combining classes are limited to the values 0 to 255.
<ul>
<li>In practice, there are far fewer than 256 values used.
Implementations may take advantage of this fact for compression, since
only the ordering of the non-zero values matters for the Canonical
Reordering Algorithm. It is possible for up to 256 values to be used
in the future; however, UTC decisions in the future may restrict the
number of values to 128, since this has implementation advantages.
[Signed bytes can be used without widening to ints in Java, for
example.]</li>
</ul>
</li>
<li>All characters other than those of General Category M* have the
combining class 0.
<ul>
<li>Currently, all characters other than those of General Category Mn
have the value 0. However, some characters of General Category Me or
Mc may be given non-zero values in the future.</li>
<li>The precise values above the value 0 are not invariant--only the
relative ordering is considered normative. For example, it is not
guaranteed in future versions that the class of U+05B4 will be
precisely 14.</li>
</ul>
</li>
</ul>
<h4>Canonical Decomposition</h4>
<ul>
<li>Canonical mappings are always in canonical order.</li>
<li>Canonical mappings have only the first of a pair possibly further
decomposing.</li>
<li>Canonical decompositions are "transparent" to other character
data:
<ul>
<li><tt>BIDI(a) = BIDI(principal(canonicalDecomposition(a))</tt></li>
<li><tt>Category(a) = Category(principal(canonicalDecomposition(a))</tt></li>
<li><tt>CombiningClass(a) =
CombiningClass(principal(canonicalDecomposition(a))</tt><br>
where principal(a) is the first character not of type Mn, or the first
character if all characters are of type Mn.</li>
</ul>
</li>
<li>However, because there are sometimes missing case pairs, and because of
some legacy characters, it is only generally true that:
<ul>
<li><tt>upper(canonicalDecomposition(a)) =
canonicalDecomposition(upper(a))</tt></li>
<li><tt>lower(canonicalDecomposition(a)) =
canonicalDecomposition(lower(a))</tt></li>
<li><tt>title(canonicalDecomposition(a)) =
canonicalDecomposition(title(a))</tt></li>
</ul>
</li>
</ul>
<h2><a name="Modification History"></a>Modification History</h2>
<p>This section provides a summary of the changes between update versions of
the Unicode Standard.</p>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 3.2">Unicode
3.2</a></h3>
<p>Modifications made for Version 3.2.0 of UnicodeData.txt include:
<blockquote>
<ul>
<li>Addition of 1016 new entries, to cover new characters encoded in
Unicode 3.2.</li>
<li>Updated ISO 6429 names for control functions to match the currently
published version of that standard.</li>
<li>Changed general category for Mongolian free variation selectors
(U+180B..U+180D) from Cf to Mn.</li>
<li>Changed general category for U+0B83 TAMIL SIGN VISARGA (aytham) from
Mc to Lo.</li>
<li>Changed general category for U+06DD ARABIC END OF AYAH from Me to Cf.</li>
<li>Changed general category for U+17D7 KHMER SIGN LEK TOO from Po to Lm.</li>
<li>Changed general category for U+17DC KHMER SIGN AVAKRAHASANYA from Po
to Lo.</li>
<li>Changed canonical decomposition for U+F951 from 96FB to 964B (see <i><a href="
http://www.unicode.org/versions/corrigendum3.html">Corrigendum
#3: U+F951 Normalization</a></i>).</li>
</ul>
</blockquote>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 3.1.1">Unicode
3.1.1</a></h3>
<p>Modifications made for Version 3.1.1 of UnicodeData.txt include:</p>
<ul>
<li>Modification of ISO 10646 annotation regarding Greek tonos, affecting
entries for U+0301 and U+030D.</li>
</ul>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 3.1">Unicode
3.1</a></h3>
<p>Modifications made for Version 3.1.0 of UnicodeData.txt include:
<ul>
<li>Addition of 2237 new entries, to cover new characters and new ranges of
unified Han characters encoded in Unicode 3.1.</li>
<li>Changed General Category value of 16EE..16F0 (Runic golden numbers) from
No to Nl.</li>
</ul>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 3.0.1">Unicode
3.0.1</a></h3>
<p>Modifications made for Version 3.0.1 of UnicodeData.txt include:
<ul>
<li>Added 5- and 6-digit representation of code points past U+FFFF.</li>
<li>Added Private Use range definitions for Planes 15 and 16.</li>
<li>Minor additions for the 10646 comment field.</li>
</ul>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 3.0.0">Unicode
3.0.0</a></h3>
<p>Modifications made for Version 3.0.0 of UnicodeData.txt include many new
characters and a number of property changes. These are summarized in Appendex
D of <em>The Unicode Standard, Version 3.0.</em></p>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 2.1.9">Unicode
2.1.9</a></h3>
<p>Modifications made for Version 2.1.9 of UnicodeData.txt include:
<ul>
<li>Corrected combining class for U+05AE HEBREW ACCENT ZINOR.</li>
<li>Corrected combining class for U+20E1 COMBINING LEFT RIGHT ARROW ABOVE</li>
<li>Corrected combining class for U+0F35 and U+0F37 to 220.</li>
<li>Corrected combining class for U+0F71 to 129.</li>
<li>Added a decomposition for U+0F0C TIBETAN MARK DELIMITER TSHEG BSTAR.</li>
<li>Added decompositions for several Greek symbol letters:
U+03D0..U+03D2, U+03D5, U+03D6, U+03F0..U+03F2.</li>
<li>Removed decompositions from the conjoining jamo block:
U+1100..U+11F8.</li>
<li>Changes to decomposition mappings for some Tibetan vowels for
consistency in normalization. (U+0F71, U+0F73, U+0F77, U+0F79, U+0F81)</li>
<li>Updated the decomposition mappings for several Vietnamese characters
with two diacritics (U+1EAC, U+1EAD, U+1EB6, U+1EB7, U+1EC6, U+1EC7,
U+1ED8, U+1ED9), so that the recursive decomposition can be generated
directly in canonically reordered form (not a normative change).</li>
<li>Updated the decomposition mappings for several Arabic compatibility
characters involving shadda (U+FC5E..U+FC62, U+FCF2..U+FCF4), and two
Latin characters (U+1E1C, U+1E1D), so that the decompositions are
generated directly in canonically reordered form (not a normative change).</li>
<li>Changed BIDI category for: U+00A0 NO-BREAK SPACE, U+2007 FIGURE SPACE,
U+2028 LINE SEPARATOR.</li>
<li>Changed BIDI category for extenders of General Category Lm: U+3005,
U+3021..U+3035, U+FF9E, U+FF9F.</li>
<li>Changed General Category and BIDI category for the Greek numeral signs:
U+0374, U+0375.</li>
<li>Corrected General Category for U+FFE8 HALFWIDTH FORMS LIGHT VERTICAL.</li>
<li>Added Unicode 1.0 names for many Tibetan characters (informative).</li>
</ul>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 2.1.8">Unicode
2.1.8</a></h3>
<p>Modifications made for Version 2.1.8 of UnicodeData.txt include:
<ul>
<li>Added combining class 240 for U+0345 COMBINING GREEK YPOGEGRAMMENI so
that decompositions involving iota subscript are derivable directly in
canonically reordered form; this also has a bearing on simplification of
casing of polytonic Greek.</li>
<li>Changes in decompositions related to Greek tonos. These result from the
clarification that monotonic Greek "tonos" should be equated
with U+0301 COMBINING ACUTE, rather than with U+030D COMBINING VERTICAL
LINE ABOVE. (All Greek characters in the Greek block involving "tonos";
some Greek characters in the polytonic Greek in the 1FXX block.)</li>
<li>Changed decompositions involving dialytika tonos. (U+0390, U+03B0)</li>
<li>Changed ternary decompositions to binary. (U+0CCB, U+FB2C, U+FB2D) These
changes simplify normalization.</li>
<li>Removed canonical decomposition for Latin Candrabindu. (U+0310)</li>
<li>Corrected error in canonical decomposition for U+1FF4.</li>
<li>Added compatibility decompositions to clarify collation tables. (U+2100,
U+2101, U+2105, U+2106, U+1E9A)</li>
<li>A series of general category changes to assist the convergence of of
Unicode definition of identifier with ISO TR 10176:
<ul>
<li>So > Lo: U+0950, U+0AD0, U+0F00, U+0F88..U+0F8B</li>
<li>Po > Lo: U+0E2F, U+0EAF, U+3006</li>
<li>Lm > Sk: U+309B, U+309C</li>
<li>Po > Pc: U+30FB, U+FF65</li>
<li>Ps/Pe > Mn: U+0F3E, U+0F3F</li>
</ul>
</li>
<li>A series of bidi property changes for consistency.
<ul>
<li>L > ET: U+09F2, U+09F3</li>
<li>ON > L: U+3007</li>
<li>L > ON: U+0F3A..U+0F3D, U+037E, U+0387</li>
</ul>
</li>
<li>Add case mapping: U+01A6 <-> U+0280</li>
<li>Updated symmetric swapping value for guillemets: U+00AB, U+00BB, U+2039,
U+203A.</li>
<li>Changes to combining class values. Most Indic fixed position class
non-spacing marks were changed to combining class 0. This fixes some
inconsistencies in how canonical reordering would apply to Indic scripts,
including Tibetan. Indic interacting top/bottom fixed position classes
were merged into single (non-zero) classes as part of this change. Tibetan
subjoined consonants are changed from combining class 6 to combining class
0. Thai pinthu (U+0E3A) moved to combining class 9. Moved two Devanagari
stress marks into generic above and below combining classes (U+0951,
U+0952).</li>
<li>Corrected placement of semicolon near symmetric swapping field. (U+FA0E,
etc., scattered positions to U+FA29)</li>
</ul>
<h3>Version 2.1.7</h3>
<p><i>This version was for internal change tracking only, and never publicly
released.</i></p>
<h3>Version 2.1.6</h3>
<p><i>This version was for internal change tracking only, and never publicly
released.</i></p>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 2.1.5">Unicode
2.1.5</a></h3>
<p>Modifications made for Version 2.1.5 of UnicodeData.txt include:
<ul>
<li>Changed decomposition for U+FF9E and U+FF9F so that correct collation
weighting will automatically result from the canonical equivalences.</li>
<li>Removed canonical decompositions for U+04D4, U+04D5, U+04D8, U+04D9,
U+04E0, U+04E1, U+04E8, U+04E9 (the implication being that no canonical
equivalence is claimed between these 8 characters and similar Latin
letters), and updated 4 canonical decompositions for U+04DB, U+04DC,
U+04EA, U+04EB to reflect the implied difference in the base character.</li>
<li>Added Pi, and Pf categories and assigned the relevant quotation marks to
those categories, based on the Unicode Technical Corrigendum on Quotation
Characters.</li>
<li>Updating of many bidi properties, following the advice of the ad hoc
committee on bidi, and to make the bidi properties of compatibility
characters more consistent.</li>
<li>Changed category of several Tibetan characters: U+0F3E, U+0F3F,
U+0F88..U+0F8B to make them non-combining, reflecting the combined opinion
of Tibetan experts.</li>
<li>Added case mapping for U+03F2.</li>
<li>Corrected case mapping for U+0275.</li>
<li>Added titlecase mappings for U+03D0, U+03D1, U+03D5, U+03D6, U+03F0..
U+03F2.</li>
<li>Corrected compatibility label for U+2121.</li>
<li>Add specific entries for all the CJK compatibility ideographs,
U+F900..U+FA2D, so the canonical decomposition for each (the URO character
it is equivalent to) can be carried in the database.</li>
</ul>
<h3>Version 2.1.4</h3>
<p><i>This version was for internal change tracking only, and never publicly
released.</i></p>
<h3>Version 2.1.3</h3>
<p><i>This version was for internal change tracking only, and never publicly
released.</i></p>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 2.1.2">Unicode
2.1.2</a></h3>
<p>Modifications made in updating UnicodeData.txt to Version 2.1.2 for the
Unicode Standard, Version 2.1 (from Version 2.0) include:
<ul>
<li>Added two characters (U+20AC and U+FFFC).</li>
<li>Amended bidi properties for U+0026, U+002E, U+0040, U+2007.</li>
<li>Corrected case mappings for U+018E, U+019F, U+01DD, U+0258, U+0275,
U+03C2, U+1E9B.</li>
<li>Changed combining order class for U+0F71.</li>
<li>Corrected canonical decompositions for U+0F73, U+1FBE.</li>
<li>Changed decomposition for U+FB1F from compatibility to canonical.</li>
<li>Added compatibility decompositions for U+FBE8, U+FBE9, U+FBF9..U+FBFB.</li>
<li>Corrected compatibility decompositions for U+2469, U+246A, U+3358.</li>
</ul>
<h3>Version 2.1.1</h3>
<p><i>This version was for internal change tracking only, and never publicly
released.</i></p>
<h3><a href="
http://www.unicode.org/unicode/standard/versions/enumeratedversions.html#Unicode 2.0.0">Unicode
2.0.0</a></h3>
<p>The modifications made in updating UnicodeData.txt for the Unicode
Standard, Version 2.0 include:
<ul>
<li>Fixed decompositions with TONOS to use correct NSM: 030D.</li>
<li>Removed old Hangul Syllables; mapping to new characters are in a
separate table.</li>
<li>Marked compatibility decompositions with additional tags.</li>
<li>Changed old tag names for clarity.</li>
<li>Revision of decompositions to use first-level decomposition, instead of
maximal decomposition.</li>
<li>Correction of all known errors in decompositions from earlier versions.</li>
<li>Added control code names (as old Unicode names).</li>
<li>Added Hangul Jamo decompositions.</li>
<li>Added Number category to match properties list in book.</li>
<li>Fixed categories of Koranic Arabic marks.</li>
<li>Fixed categories of precomposed characters to match decomposition where
possible.</li>
<li>Added Hebrew cantillation marks and the Tibetan script.</li>
<li>Added place holders for ranges such as CJK Ideographic Area and the
Private Use Area.</li>
<li>Added categories Me, Sk, Pc, Nl, Cs, Cf, and rectified a number of
mistakes in the database.</li>
</ul>
<h2><i><a name="UCD_Terms">UCD Terms of Use</a></i></h2>
<h3><i>Disclaimer</i></h3>
<blockquote>
<p><i>The Unicode Character Database is provided as is by Unicode, Inc. No
claims are made as to fitness for any particular purpose. No warranties of
any kind are expressed or implied. The recipient agrees to determine
applicability of information provided. If this file has been purchased on
magnetic or optical media from Unicode, Inc., the sole remedy for any claim
will be exchange of defective media within 90 days of receipt.</i></p>
<p><i>This disclaimer is applicable for all other data files accompanying
the Unicode Character Database, some of which have been compiled by the
Unicode Consortium, and some of which have been supplied by other sources.</i></p>
</blockquote>
<h3><i>Limitations on Rights to Redistribute This Data</i></h3>
<blockquote>
<p><i>Recipient is granted the right to make copies in any form for internal
distribution and to freely use the information supplied in the creation of
products supporting the Unicode<sup>TM</sup> Standard. The files in the
Unicode Character Database can be redistributed to third parties or other
organizations (whether for profit or not) as long as this notice and the
disclaimer notice are retained. Information can be extracted from these
files and used in documentation or programs, as long as there is an
accompanying notice indicating the source.</i></p>
</blockquote>
<hr width="50%">
<div align="center">
<center>
<table>
<tr>
<td><a href="
http://www.unicode.org/unicode/copyright.html"><img src="
http://www.unicode.org/img/hb_home.gif" border="0" alt="Home" width="40" height="49"><img src="
http://www.unicode.org/img/hb_mid.gif" border="0" alt="Terms of Use" width="152" height="49"><img src="
http://www.unicode.org/img/hb_mail.gif" border="0" alt="E-mail" width="46" height="49"></a></td>
</tr>
</table>
</center>
</div>
</blockquote>
</body>
</html>
