Issue #4153: Updated Unicode HOWTO.

abalkin · abalkin · commit 93a6b13f966a · 2010-11-19T16:09:58.000Z
diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst
@@ -4,13 +4,11 @@
   Unicode HOWTO
 *****************
 
-:Release: 1.11
+:Release: 1.12
 
-This HOWTO discusses Python 2.x's support for Unicode, and explains
+This HOWTO discusses Python support for Unicode, and explains
 various problems that people commonly encounter when trying to work
-with Unicode.  (This HOWTO has not yet been updated to cover the 3.x
-versions of Python.)
-
+with Unicode.
 
 Introduction to Unicode
 =======================
@@ -44,14 +42,14 @@ In the 1980s, almost all personal computers were 8-bit, meaning that bytes could
 hold values ranging from 0 to 255.  ASCII codes only went up to 127, so some
 machines assigned values between 128 and 255 to accented characters.  Different
 machines had different codes, however, which led to problems exchanging files.
-Eventually various commonly used sets of values for the 128-255 range emerged.
+Eventually various commonly used sets of values for the 128--255 range emerged.
 Some were true standards, defined by the International Standards Organization,
 and some were **de facto** conventions that were invented by one company or
 another and managed to catch on.
 
 255 characters aren't very many.  For example, you can't fit both the accented
 characters used in Western Europe and the Cyrillic alphabet used for Russian
-into the 128-255 range because there are more than 127 such characters.
+into the 128--255 range because there are more than 127 such characters.
 
 You could write files using different codes (all your Russian files in a coding
 system called KOI8, all your French files in a different coding system called
@@ -64,8 +62,8 @@ bits means you have 2^16 = 65,536 distinct values available, making it possible
 to represent many different characters from many different alphabets; an initial
 goal was to have Unicode contain the alphabets for every single human language.
 It turns out that even 16 bits isn't enough to meet that goal, and the modern
-Unicode specification uses a wider range of codes, 0-1,114,111 (0x10ffff in
-base-16).
+Unicode specification uses a wider range of codes, 0 through 1,114,111 (0x10ffff
+in base 16).
 
 There's a related ISO standard, ISO 10646.  Unicode and ISO 10646 were
 originally separate efforts, but the specifications were merged with the 1.1
@@ -90,7 +88,7 @@ meanings.
 The Unicode standard describes how characters are represented by **code
 points**.  A code point is an integer value, usually denoted in base 16.  In the
 standard, a code point is written using the notation U+12ca to mean the
-character with value 0x12ca (4810 decimal).  The Unicode standard contains a lot
+character with value 0x12ca (4,810 decimal).  The Unicode standard contains a lot
 of tables listing characters and their corresponding code points::
 
    0061    'a'; LATIN SMALL LETTER A
@@ -117,10 +115,10 @@ Encodings
 ---------
 
 To summarize the previous section: a Unicode string is a sequence of code
-points, which are numbers from 0 to 0x10ffff.  This sequence needs to be
-represented as a set of bytes (meaning, values from 0-255) in memory.  The rules
-for translating a Unicode string into a sequence of bytes are called an
-**encoding**.
+points, which are numbers from 0 through 0x10ffff (1,114,111 decimal).  This
+sequence needs to be represented as a set of bytes (meaning, values
+from 0 through 255) in memory.  The rules for translating a Unicode string
+into a sequence of bytes are called an **encoding**.
 
 The first encoding you might think of is an array of 32-bit integers.  In this
 representation, the string "Python" would look like this::
@@ -164,7 +162,7 @@ encoding, for example, are simple; for each code point:
    case.)
 
 Latin-1, also known as ISO-8859-1, is a similar encoding.  Unicode code points
-0-255 are identical to the Latin-1 values, so converting to this encoding simply
+0--255 are identical to the Latin-1 values, so converting to this encoding simply
 requires converting code points to byte values; if a code point larger than 255
 is encountered, the string can't be encoded into Latin-1.
 
@@ -226,8 +224,8 @@ Wikipedia entries are often helpful; see the entries for "character encoding"
 <http://en.wikipedia.org/wiki/UTF-8>, for example.
 
 
-Python 2.x's Unicode Support
-============================
+Python's Unicode Support
+========================
 
 Now that you've learned the rudiments of Unicode, we can look at Python's
 Unicode features.
@@ -265,7 +263,7 @@ Unicode result).  The following examples show the differences::
     UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0:
                         unexpected code byte
     >>> b'\x80abc'.decode("utf-8", "replace")
-    '\ufffdabc'
+    '�abc'
     >>> b'\x80abc'.decode("utf-8", "ignore")
     'abc'
 
@@ -281,10 +279,10 @@ that contains the corresponding code point.  The reverse operation is the
 built-in :func:`ord` function that takes a one-character Unicode string and
 returns the code point value::
 
-    >>> chr(40960)
-    '\ua000'
-    >>> ord('\ua000')
-    40960
+    >>> chr(57344)
+    '\ue000'
+    >>> ord('\ue000')
+    57344
 
 Converting to Bytes
 -------------------
@@ -326,7 +324,8 @@ Unicode Literals in Python Source Code
 
 In Python source code, specific Unicode code points can be written using the
 ``\u`` escape sequence, which is followed by four hex digits giving the code
-point.  The ``\U`` escape sequence is similar, but expects 8 hex digits, not 4::
+point.  The ``\U`` escape sequence is similar, but expects eight hex digits,
+not four::
 
     >>> s = "a\xac\u1234\u20ac\U00008000"
               ^^^^ two-digit hex escape
@@ -465,18 +464,17 @@ like those in string objects' :meth:`encode` and :meth:`decode` methods.
 
 Reading Unicode from a file is therefore simple::
 
-    f = open('unicode.rst', encoding='utf-8')
-    for line in f:
-        print(repr(line))
+    with open('unicode.rst', encoding='utf-8') as f:
+        for line in f:
+            print(repr(line))
 
 It's also possible to open files in update mode, allowing both reading and
 writing::
 
-    f = open('test', encoding='utf-8', mode='w+')
-    f.write('\u4500 blah blah blah\n')
-    f.seek(0)
-    print(repr(f.readline()[:1]))
-    f.close()
+    with open('test', encoding='utf-8', mode='w+') as f:
+        f.write('\u4500 blah blah blah\n')
+        f.seek(0)
+        print(repr(f.readline()[:1]))
 
 The Unicode character U+FEFF is used as a byte-order mark (BOM), and is often
 written as the first character of a file in order to assist with autodetection
@@ -513,14 +511,13 @@ usually just provide the Unicode string as the filename, and it will be
 automatically converted to the right encoding for you::
 
     filename = 'filename\u4500abc'
-    f = open(filename, 'w')
-    f.write('blah\n')
-    f.close()
+    with open(filename, 'w') as f:
+        f.write('blah\n')
 
 Functions in the :mod:`os` module such as :func:`os.stat` will also accept Unicode
 filenames.
 
-:func:`os.listdir`, which returns filenames, raises an issue: should it return
+Function :func:`os.listdir`, which returns filenames, raises an issue: should it return
 the Unicode version of filenames, or should it return byte strings containing
 the encoded versions?  :func:`os.listdir` will do both, depending on whether you
 provided the directory path as a byte string or a Unicode string.  If you pass a
@@ -569,14 +566,6 @@ strings, you will find your program vulnerable to bugs wherever you combine the
 two different kinds of strings.  There is no automatic encoding or decoding if
 you do e.g. ``str + bytes``, a :exc:`TypeError` is raised for this expression.
 
-It's easy to miss such problems if you only test your software with data that
-doesn't contain any accents; everything will seem to work, but there's actually
-a bug in your program waiting for the first user who attempts to use characters
-> 127.  A second tip, therefore, is:
-
-    Include characters > 127 and, even better, characters > 255 in your test
-    data.
-
 When using data coming from a web browser or some other untrusted source, a
 common technique is to check for illegal characters in a string before using the
 string in a generated command line or storing it in a database.  If you're doing
@@ -594,8 +583,8 @@ this code::
         if '/' in filename:
             raise ValueError("'/' not allowed in filenames")
         unicode_name = filename.decode(encoding)
-        f = open(unicode_name, 'r')
-        # ... return contents of file ...
+        with open(unicode_name, 'r') as f:
+            # ... return contents of file ...
 
 However, if an attacker could specify the ``'base64'`` encoding, they could pass
 ``'L2V0Yy9wYXNzd2Q='``, which is the base-64 encoded form of the string
@@ -610,27 +599,30 @@ The PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware
 Applications in Python" are available at
 <http://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>
 and discuss questions of character encodings as well as how to internationalize
-and localize an application.
+and localize an application.  These slides cover Python 2.x only.
 
 
-Revision History and Acknowledgements
-=====================================
+Acknowledgements
+================
 
 Thanks to the following people who have noted errors or offered suggestions on
 this article: Nicholas Bastin, Marius Gedminas, Kent Johnson, Ken Krugler,
 Marc-André Lemburg, Martin von Löwis, Chad Whitacre.
 
-Version 1.0: posted August 5 2005.
+.. comment
+   Revision History
+
+   Version 1.0: posted August 5 2005.
 
-Version 1.01: posted August 7 2005.  Corrects factual and markup errors; adds
-several links.
+   Version 1.01: posted August 7 2005.  Corrects factual and markup errors; adds
+   several links.
 
-Version 1.02: posted August 16 2005.  Corrects factual errors.
+   Version 1.02: posted August 16 2005.  Corrects factual errors.
 
-Version 1.1: Feb-Nov 2008.  Updates the document with respect to Python 3 changes.
+   Version 1.1: Feb-Nov 2008.  Updates the document with respect to Python 3 changes.
 
-Version 1.11: posted June 20 2010.  Notes that Python 3.x is not covered,
-and that the HOWTO only covers 2.x.
+   Version 1.11: posted June 20 2010.  Notes that Python 3.x is not covered,
+   and that the HOWTO only covers 2.x.
 
 .. comment Describe Python 3.x support (new section? new document?)
 .. comment Additional topic: building Python w/ UCS2 or UCS4 support
