summaryrefslogtreecommitdiffstats
path: root/man7/utf-8.7
diff options
context:
space:
mode:
Diffstat (limited to 'man7/utf-8.7')
-rw-r--r--man7/utf-8.7211
1 files changed, 211 insertions, 0 deletions
diff --git a/man7/utf-8.7 b/man7/utf-8.7
new file mode 100644
index 0000000..8a5f7ab
--- /dev/null
+++ b/man7/utf-8.7
@@ -0,0 +1,211 @@
+.\" Copyright (C) Markus Kuhn, 1996, 2001
+.\"
+.\" SPDX-License-Identifier: GPL-2.0-or-later
+.\"
+.\" 1995-11-26 Markus Kuhn <mskuhn@cip.informatik.uni-erlangen.de>
+.\" First version written
+.\" 2001-05-11 Markus Kuhn <mgk25@cl.cam.ac.uk>
+.\" Update
+.\"
+.TH UTF-8 7 2023-03-12 "Linux man-pages 6.05.01"
+.SH NAME
+UTF-8 \- an ASCII compatible multibyte Unicode encoding
+.SH DESCRIPTION
+The Unicode 3.0 character set occupies a 16-bit code space.
+The most obvious
+Unicode encoding (known as UCS-2)
+consists of a sequence of 16-bit words.
+Such strings can contain\[em]as part of many 16-bit characters\[em]bytes
+such as \[aq]\e0\[aq] or \[aq]/\[aq], which have a
+special meaning in filenames and other C library function arguments.
+In addition, the majority of UNIX tools expect ASCII files and can't
+read 16-bit words as characters without major modifications.
+For these reasons,
+UCS-2 is not a suitable external encoding of Unicode
+in filenames, text files, environment variables, and so on.
+The ISO/IEC 10646 Universal Character Set (UCS),
+a superset of Unicode, occupies an even larger code
+space\[em]31\ bits\[em]and the obvious
+UCS-4 encoding for it (a sequence of 32-bit words) has the same problems.
+.PP
+The UTF-8 encoding of Unicode and UCS
+does not have these problems and is the common way in which
+Unicode is used on UNIX-style operating systems.
+.SS Properties
+The UTF-8 encoding has the following nice properties:
+.TP 0.2i
+*
+UCS
+characters 0x00000000 to 0x0000007f (the classic US-ASCII
+characters) are encoded simply as bytes 0x00 to 0x7f (ASCII
+compatibility).
+This means that files and strings which contain only
+7-bit ASCII characters have the same encoding under both
+ASCII
+and
+UTF-8 .
+.TP
+*
+All UCS characters greater than 0x7f are encoded as a multibyte sequence
+consisting only of bytes in the range 0x80 to 0xfd, so no ASCII
+byte can appear as part of another character and there are no
+problems with, for example, \[aq]\e0\[aq] or \[aq]/\[aq].
+.TP
+*
+The lexicographic sorting order of UCS-4 strings is preserved.
+.TP
+*
+All possible 2\[ha]31 UCS codes can be encoded using UTF-8.
+.TP
+*
+The bytes 0xc0, 0xc1, 0xfe, and 0xff are never used in the UTF-8 encoding.
+.TP
+*
+The first byte of a multibyte sequence which represents a single non-ASCII
+UCS character is always in the range 0xc2 to 0xfd and indicates how long
+this multibyte sequence is.
+All further bytes in a multibyte sequence
+are in the range 0x80 to 0xbf.
+This allows easy resynchronization and
+makes the encoding stateless and robust against missing bytes.
+.TP
+*
+UTF-8 encoded UCS characters may be up to six bytes long, however the
+Unicode standard specifies no characters above 0x10ffff, so Unicode characters
+can be only up to four bytes long in
+UTF-8.
+.SS Encoding
+The following byte sequences are used to represent a character.
+The sequence to be used depends on the UCS code number of the character:
+.TP 0.4i
+0x00000000 \- 0x0000007F:
+.RI 0 xxxxxxx
+.TP
+0x00000080 \- 0x000007FF:
+.RI 110 xxxxx
+.RI 10 xxxxxx
+.TP
+0x00000800 \- 0x0000FFFF:
+.RI 1110 xxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.TP
+0x00010000 \- 0x001FFFFF:
+.RI 11110 xxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.TP
+0x00200000 \- 0x03FFFFFF:
+.RI 111110 xx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.TP
+0x04000000 \- 0x7FFFFFFF:
+.RI 1111110 x
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.RI 10 xxxxxx
+.PP
+The
+.I xxx
+bit positions are filled with the bits of the character code number in
+binary representation, most significant bit first (big-endian).
+Only the shortest possible multibyte sequence
+which can represent the code number of the character can be used.
+.PP
+The UCS code values 0xd800\[en]0xdfff (UTF-16 surrogates) as well as 0xfffe and
+0xffff (UCS noncharacters) should not appear in conforming UTF-8 streams.
+According to RFC 3629 no point above U+10FFFF should be used,
+which limits characters to four bytes.
+.SS Example
+The Unicode character 0xa9 = 1010 1001 (the copyright sign) is encoded
+in UTF-8 as
+.PP
+.RS
+11000010 10101001 = 0xc2 0xa9
+.RE
+.PP
+and character 0x2260 = 0010 0010 0110 0000 (the "not equal" symbol) is
+encoded as:
+.PP
+.RS
+11100010 10001001 10100000 = 0xe2 0x89 0xa0
+.RE
+.SS Application notes
+Users have to select a UTF-8 locale, for example with
+.PP
+.RS
+export LANG=en_GB.UTF-8
+.RE
+.PP
+in order to activate the UTF-8 support in applications.
+.PP
+Application software that has to be aware of the used character
+encoding should always set the locale with for example
+.PP
+.RS
+setlocale(LC_CTYPE, "")
+.RE
+.PP
+and programmers can then test the expression
+.PP
+.RS
+strcmp(nl_langinfo(CODESET), "UTF-8") == 0
+.RE
+.PP
+to determine whether a UTF-8 locale has been selected and whether
+therefore all plaintext standard input and output, terminal
+communication, plaintext file content, filenames, and environment
+variables are encoded in UTF-8.
+.PP
+Programmers accustomed to single-byte encodings such as US-ASCII or ISO 8859
+have to be aware that two assumptions made so far are no longer valid
+in UTF-8 locales.
+Firstly, a single byte does not necessarily correspond any
+more to a single character.
+Secondly, since modern terminal emulators in UTF-8
+mode also support Chinese, Japanese, and Korean
+double-width characters as well as nonspacing combining characters,
+outputting a single character does not necessarily advance the cursor
+by one position as it did in ASCII.
+Library functions such as
+.BR mbsrtowcs (3)
+and
+.BR wcswidth (3)
+should be used today to count characters and cursor positions.
+.PP
+The official ESC sequence to switch from an ISO 2022
+encoding scheme (as used for instance by VT100 terminals) to
+UTF-8 is ESC % G
+("\ex1b%G").
+The corresponding return sequence from
+UTF-8 to ISO 2022 is ESC % @ ("\ex1b%@").
+Other ISO 2022 sequences (such as
+for switching the G0 and G1 sets) are not applicable in UTF-8 mode.
+.SS Security
+The Unicode and UCS standards require that producers of UTF-8
+shall use the shortest form possible, for example, producing a two-byte
+sequence with first byte 0xc0 is nonconforming.
+Unicode 3.1 has added the requirement that conforming programs must not accept
+non-shortest forms in their input.
+This is for security reasons: if
+user input is checked for possible security violations, a program
+might check only for the ASCII
+version of "/../" or ";" or NUL and overlook that there are many
+non-ASCII ways to represent these things in a non-shortest UTF-8
+encoding.
+.SS Standards
+ISO/IEC 10646-1:2000, Unicode 3.1, RFC\ 3629, Plan 9.
+.\" .SH AUTHOR
+.\" Markus Kuhn <mgk25@cl.cam.ac.uk>
+.SH SEE ALSO
+.BR locale (1),
+.BR nl_langinfo (3),
+.BR setlocale (3),
+.BR charsets (7),
+.BR unicode (7)