For a while people just wrote programs that didn't display accents.

In the mid-1980s an Apple II BASIC program written by a French speaker

might have lines like these::

Those messages should contain accents (completé, caractère, accepté),

and they just look wrong to someone who can read French.

(This discussion of Unicode's history is highly simplified. The

precise historical details aren't necessary for understanding how to

use Unicode effectively, but if you're curious, consult the Unicode

consortium site listed in the References or

the `Wikipedia entry for Unicode <http://en.wikipedia.org/wiki/Unicode#History>`_

for more information.)

If this introduction didn't make things clear to you, you should try

reading this alternate article before continuing.

The default encoding for Python source code is UTF-8, so you can simply

include a Unicode character in a string literal::

You can use a different encoding from UTF-8 by putting a specially-formatted

comment as the first or second line of the source code::

Side note: Python 3 also supports using Unicode characters in identifiers::

If you can't enter a particular character in your editor or want to

keep the source code ASCII-only for some reason, you can also use

escape sequences in string literals. (Depending on your system,

you may see the actual capital-delta glyph instead of a \u escape.) ::

and optionally an *errors* argument.

requested *encoding*.

The *errors* parameter is the same as the parameter of the

:meth:`~bytes.decode` method but supports a few more possible handlers. As well as

``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case

inserts a question mark instead of the unencodable character), there is

also ``'xmlcharrefreplace'`` (inserts an XML character reference) and

``backslashreplace`` (inserts a ``\uNNNN`` escape sequence).

The low-level routines for registering and accessing the available

encodings are found in the :mod:`codecs` module. Implementing new

encodings also requires understanding the :mod:`codecs` module.

However, the encoding and decoding functions returned by this module

are usually more low-level than is comfortable, and writing new encodings

is a specialized task, so the module won't be covered in this HOWTO.

`the General Category Values section of the Unicode Character Database documentation <http://www.unicode.org/reports/tr44/#General_Category_Values>`_ for a

Unicode Regular Expressions

The regular expressions supported by the :mod:`re` module can be provided

either as bytes or strings. Some of the special character sequences such as

``\d`` and ``\w`` have different meanings depending on whether

the pattern is supplied as bytes or a string. For example,

``\d`` will match the characters ``[0-9]`` in bytes but

in strings will match any character that's in the ``'Nd'`` category.

The string in this example has the number 57 written in both Thai and

Arabic numerals::

When executed, ``\d+`` will match the Thai numerals and print them

out. If you supply the :const:`re.ASCII` flag to

:func:`~re.compile`, ``\d+`` will match the substring "57" instead.

Similarly, ``\w`` matches a wide variety of Unicode characters but

only ``[a-zA-Z0-9_]`` in bytes or if :const:`re.ASCII` is supplied,

and ``\s`` will match either Unicode whitespace characters or

``[ \t\n\r\f\v]``.

Some good alternative discussions of Python's Unicode support are:

* `Processing Text Files in Python 3 <http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html>`_, by Nick Coghlan.

* `Pragmatic Unicode <http://nedbatchelder.com/text/unipain.html>`_, a PyCon 2012 presentation by Ned Batchelder.

Marc-André Lemburg gave `a presentation titled "Python and Unicode" (PDF slides) <http://downloads.egenix.com/python/Unicode-EPC2002-Talk.pdf>`_ at

EuroPython 2002. The slides are an excellent overview of the design

of Python 2's Unicode features (where the Unicode string type is

called ``unicode`` and literals start with ``u``).

you haven't, the default encoding is UTF-8.

The :func:`os.listdir` function returns filenames and raises an issue: should it return

path will return the filenames as bytes. For example,

Note that on most occasions, the Unicode APIs should be used. The bytes APIs

Converting Between File Encodings

The :class:`~codecs.StreamRecoder` class can transparently convert between

encodings, taking a stream that returns data in encoding #1

and behaving like a stream returning data in encoding #2.

For example, if you have an input file *f* that's in Latin-1, you

can wrap it with a :class:`StreamRecoder` to return bytes encoded in UTF-8::

Files in an Unknown Encoding

What can you do if you need to make a change to a file, but don't know

the file's encoding? If you know the encoding is ASCII-compatible and

only want to examine or modify the ASCII parts, you can open the file

with the ``surrogateescape`` error handler::

The ``surrogateescape`` error handler will decode any non-ASCII bytes

as code points in the Unicode Private Use Area ranging from U+DC80 to

U+DCFF. These private code points will then be turned back into the

same bytes when the ``surrogateescape`` error handler is used when

encoding the data and writing it back out.

The `PDF slides for Marc-André Lemburg's presentation "Writing Unicode-aware Applications in Python" <http://downloads.egenix.com/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>`_

discuss questions of character encodings as well as how to internationalize

`The Guts of Unicode in Python <http://pyvideo.org/video/1768/the-guts-of-unicode-in-python>`_ is a PyCon 2013 talk by Benjamin Peterson that discusses the internal Unicode representation in Python 3.3.

The initial draft of this document was written by Andrew Kuchling.

It has since been revised further by Alexander Belopolsky, Georg Brandl,

Andrew Kuchling, and Ezio Melotti.

Thanks to the following people who have noted errors or offered

suggestions on this article: Éric Araujo, Nicholas Bastin, Nick

Coghlan, Marius Gedminas, Kent Johnson, Ken Krugler, Marc-André

Lemburg, Martin von Löwis, Terry J. Reedy, Chad Whitacre.