Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 3929499

Browse files
committed
Issue #1602: Windows console doesn't input or print Unicode (PEP 528)
Closes #17602: Adds a readline implementation for the Windows console
1 parent b957b0c commit 3929499

16 files changed

Lines changed: 1739 additions & 21 deletions

File tree

Doc/library/functions.rst

Lines changed: 24 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1055,30 +1055,38 @@ are always available. They are listed here in alphabetical order.
10551055
(where :func:`open` is declared), :mod:`os`, :mod:`os.path`, :mod:`tempfile`,
10561056
and :mod:`shutil`.
10571057

1058-
.. versionchanged:: 3.3
1059-
The *opener* parameter was added.
1060-
The ``'x'`` mode was added.
1061-
:exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`.
1062-
:exc:`FileExistsError` is now raised if the file opened in exclusive
1063-
creation mode (``'x'``) already exists.
1058+
.. versionchanged::
1059+
3.3
10641060

1065-
.. versionchanged:: 3.4
1066-
The file is now non-inheritable.
1061+
* The *opener* parameter was added.
1062+
* The ``'x'`` mode was added.
1063+
* :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`.
1064+
* :exc:`FileExistsError` is now raised if the file opened in exclusive
1065+
* creation mode (``'x'``) already exists.
1066+
1067+
.. versionchanged::
1068+
3.4
1069+
1070+
* The file is now non-inheritable.
10671071

10681072
.. deprecated-removed:: 3.4 4.0
10691073

10701074
The ``'U'`` mode.
10711075

1072-
.. versionchanged:: 3.5
1073-
If the system call is interrupted and the signal handler does not raise an
1074-
exception, the function now retries the system call instead of raising an
1075-
:exc:`InterruptedError` exception (see :pep:`475` for the rationale).
1076+
.. versionchanged::
1077+
3.5
10761078

1077-
.. versionchanged:: 3.5
1078-
The ``'namereplace'`` error handler was added.
1079+
* If the system call is interrupted and the signal handler does not raise an
1080+
exception, the function now retries the system call instead of raising an
1081+
:exc:`InterruptedError` exception (see :pep:`475` for the rationale).
1082+
* The ``'namereplace'`` error handler was added.
10791083

1080-
.. versionchanged:: 3.6
1081-
Support added to accept objects implementing :class:`os.PathLike`.
1084+
.. versionchanged::
1085+
3.6
1086+
1087+
* Support added to accept objects implementing :class:`os.PathLike`.
1088+
* On Windows, opening a console buffer may return a subclass of
1089+
:class:`io.RawIOBase` other than :class:`io.FileIO`.
10821090

10831091
.. function:: ord(c)
10841092

Doc/using/cmdline.rst

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -559,6 +559,10 @@ conflict.
559559
.. versionchanged:: 3.4
560560
The ``encodingname`` part is now optional.
561561

562+
.. versionchanged:: 3.6
563+
On Windows, the encoding specified by this variable is ignored for interactive
564+
console buffers unless :envvar:`PYTHONLEGACYWINDOWSIOENCODING` is also specified.
565+
Files and pipes redirected through the standard streams are not affected.
562566

563567
.. envvar:: PYTHONNOUSERSITE
564568

@@ -686,6 +690,19 @@ conflict.
686690
.. versionadded:: 3.6
687691
See :pep:`529` for more details.
688692

693+
.. envvar:: PYTHONLEGACYWINDOWSIOENCODING
694+
695+
If set to a non-empty string, does not use the new console reader and
696+
writer. This means that Unicode characters will be encoded according to
697+
the active console code page, rather than using utf-8.
698+
699+
This variable is ignored if the standard streams are redirected (to files
700+
or pipes) rather than referring to console buffers.
701+
702+
Availability: Windows
703+
704+
.. versionadded:: 3.6
705+
689706
Debug-mode variables
690707
~~~~~~~~~~~~~~~~~~~~
691708

Doc/whatsnew/3.6.rst

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ Windows improvements:
7878

7979
* PEP 529: :ref:`Change Windows filesystem encoding to UTF-8 <pep-529>`
8080

81+
* PEP 528: :ref:`Change Windows console encoding to UTF-8 <pep-528>`
82+
8183
* The ``py.exe`` launcher, when used interactively, no longer prefers
8284
Python 2 over Python 3 when the user doesn't specify a version (via
8385
command line arguments or a config file). Handling of shebang lines
@@ -267,6 +269,23 @@ Also see :pep:`487` and the updated class customization documentation at
267269

268270
(Contributed by Martin Teichmann in :issue:`27366`)
269271

272+
.. _pep-528:
273+
274+
PEP 528: Change Windows console encoding to UTF-8
275+
-------------------------------------------------
276+
277+
The default console on Windows will now accept all Unicode characters and
278+
provide correctly read str objects to Python code. ``sys.stdin``,
279+
``sys.stdout`` and ``sys.stderr`` now default to utf-8 encoding.
280+
281+
This change only applies when using an interactive console, and not when
282+
redirecting files or pipes. To revert to the previous behaviour for interactive
283+
console use, set :envvar:`PYTHONLEGACYWINDOWSIOENCODING`.
284+
285+
.. seealso::
286+
287+
:pep:`528` -- Change Windows console encoding to UTF-8
288+
PEP written and implemented by Steve Dower.
270289

271290
PYTHONMALLOC environment variable
272291
=================================

Include/pydebug.h

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ PyAPI_DATA(int) Py_UnbufferedStdioFlag;
2424
PyAPI_DATA(int) Py_HashRandomizationFlag;
2525
PyAPI_DATA(int) Py_IsolatedFlag;
2626

27+
#ifdef MS_WINDOWS
28+
PyAPI_DATA(int) Py_LegacyWindowsStdioFlag;
29+
#endif
30+
2731
/* this is a wrapper around getenv() that pays attention to
2832
Py_IgnoreEnvironmentFlag. It should be used for getting variables like
2933
PYTHONPATH and PYTHONHOME from the environment */

Lib/io.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,3 +90,10 @@ class TextIOBase(_io._TextIOBase, IOBase):
9090
for klass in (StringIO, TextIOWrapper):
9191
TextIOBase.register(klass)
9292
del klass
93+
94+
try:
95+
from _io import _WindowsConsoleIO
96+
except ImportError:
97+
pass
98+
else:
99+
RawIOBase.register(_WindowsConsoleIO)

Lib/test/test_os.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1518,7 +1518,7 @@ class TestInvalidFD(unittest.TestCase):
15181518
singles = ["fchdir", "dup", "fdopen", "fdatasync", "fstat",
15191519
"fstatvfs", "fsync", "tcgetpgrp", "ttyname"]
15201520
#singles.append("close")
1521-
#We omit close because it doesn'r raise an exception on some platforms
1521+
#We omit close because it doesn't raise an exception on some platforms
15221522
def get_single(f):
15231523
def helper(self):
15241524
if hasattr(os, f):

Lib/test/test_winconsoleio.py

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
'''Tests for WindowsConsoleIO
2+
3+
Unfortunately, most testing requires interactive use, since we have no
4+
API to read back from a real console, and this class is only for use
5+
with real consoles.
6+
7+
Instead, we validate that basic functionality such as opening, closing
8+
and in particular fileno() work, but are forced to leave real testing
9+
to real people with real keyborads.
10+
'''
11+
12+
import io
13+
import unittest
14+
import sys
15+
16+
if sys.platform != 'win32':
17+
raise unittest.SkipTest("test only relevant on win32")
18+
19+
ConIO = io._WindowsConsoleIO
20+
21+
class WindowsConsoleIOTests(unittest.TestCase):
22+
def test_abc(self):
23+
self.assertTrue(issubclass(ConIO, io.RawIOBase))
24+
self.assertFalse(issubclass(ConIO, io.BufferedIOBase))
25+
self.assertFalse(issubclass(ConIO, io.TextIOBase))
26+
27+
def test_open_fd(self):
28+
f = ConIO(0)
29+
self.assertTrue(f.readable())
30+
self.assertFalse(f.writable())
31+
self.assertEqual(0, f.fileno())
32+
f.close() # multiple close should not crash
33+
f.close()
34+
35+
f = ConIO(1, 'w')
36+
self.assertFalse(f.readable())
37+
self.assertTrue(f.writable())
38+
self.assertEqual(1, f.fileno())
39+
f.close()
40+
f.close()
41+
42+
f = ConIO(2, 'w')
43+
self.assertFalse(f.readable())
44+
self.assertTrue(f.writable())
45+
self.assertEqual(2, f.fileno())
46+
f.close()
47+
f.close()
48+
49+
def test_open_name(self):
50+
f = ConIO("CON")
51+
self.assertTrue(f.readable())
52+
self.assertFalse(f.writable())
53+
self.assertIsNotNone(f.fileno())
54+
f.close() # multiple close should not crash
55+
f.close()
56+
57+
f = ConIO('CONIN$')
58+
self.assertTrue(f.readable())
59+
self.assertFalse(f.writable())
60+
self.assertIsNotNone(f.fileno())
61+
f.close()
62+
f.close()
63+
64+
f = ConIO('CONOUT$', 'w')
65+
self.assertFalse(f.readable())
66+
self.assertTrue(f.writable())
67+
self.assertIsNotNone(f.fileno())
68+
f.close()
69+
f.close()
70+
71+
if __name__ == "__main__":
72+
unittest.main()

Misc/NEWS

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,8 @@ Build
300300
Windows
301301
-------
302302

303+
- Issue #1602: Windows console doesn't input or print Unicode (PEP 528)
304+
303305
- Issue #27781: Change file system encoding on Windows to UTF-8 (PEP 529)
304306

305307
- Issue #27731: Opt-out of MAX_PATH on Windows 10
@@ -556,7 +558,6 @@ Build
556558
- Issue #10910: Avoid C++ compilation errors on FreeBSD and OS X.
557559
Also update FreedBSD version checks for the original ctype UTF-8 workaround.
558560

559-
560561
What's New in Python 3.6.0 alpha 3
561562
==================================
562563

Modules/_io/_iomodule.c

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,9 @@
2020
#include <sys/stat.h>
2121
#endif /* HAVE_SYS_STAT_H */
2222

23+
#ifdef MS_WINDOWS
24+
#include <consoleapi.h>
25+
#endif
2326

2427
/* Various interned strings */
2528

@@ -52,7 +55,6 @@ PyObject *_PyIO_empty_str;
5255
PyObject *_PyIO_empty_bytes;
5356
PyObject *_PyIO_zero;
5457

55-
5658
PyDoc_STRVAR(module_doc,
5759
"The io module provides the Python interfaces to stream handling. The\n"
5860
"builtin open function is defined in this module.\n"
@@ -362,8 +364,18 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode,
362364
}
363365

364366
/* Create the Raw file stream */
365-
raw = PyObject_CallFunction((PyObject *)&PyFileIO_Type,
366-
"OsiO", path_or_fd, rawmode, closefd, opener);
367+
{
368+
PyObject *RawIO_class = (PyObject *)&PyFileIO_Type;
369+
#ifdef MS_WINDOWS
370+
if (!Py_LegacyWindowsStdioFlag && _PyIO_get_console_type(path_or_fd) != '\0') {
371+
RawIO_class = (PyObject *)&PyWindowsConsoleIO_Type;
372+
encoding = "utf-8";
373+
}
374+
#endif
375+
raw = PyObject_CallFunction(RawIO_class,
376+
"OsiO", path_or_fd, rawmode, closefd, opener);
377+
}
378+
367379
if (raw == NULL)
368380
goto error;
369381
result = raw;
@@ -708,6 +720,12 @@ PyInit__io(void)
708720
PyStringIO_Type.tp_base = &PyTextIOBase_Type;
709721
ADD_TYPE(&PyStringIO_Type, "StringIO");
710722

723+
#ifdef MS_WINDOWS
724+
/* WindowsConsoleIO */
725+
PyWindowsConsoleIO_Type.tp_base = &PyRawIOBase_Type;
726+
ADD_TYPE(&PyWindowsConsoleIO_Type, "_WindowsConsoleIO");
727+
#endif
728+
711729
/* BufferedReader */
712730
PyBufferedReader_Type.tp_base = &PyBufferedIOBase_Type;
713731
ADD_TYPE(&PyBufferedReader_Type, "BufferedReader");

Modules/_io/_iomodule.h

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,12 @@ extern PyTypeObject PyBufferedRandom_Type;
1919
extern PyTypeObject PyTextIOWrapper_Type;
2020
extern PyTypeObject PyIncrementalNewlineDecoder_Type;
2121

22+
#ifndef Py_LIMITED_API
23+
#ifdef MS_WINDOWS
24+
extern PyTypeObject PyWindowsConsoleIO_Type;
25+
#define PyWindowsConsoleIO_Check(op) (PyObject_TypeCheck((op), &PyWindowsConsoleIO_Type))
26+
#endif /* MS_WINDOWS */
27+
#endif /* Py_LIMITED_API */
2228

2329
extern int _PyIO_ConvertSsize_t(PyObject *, void *);
2430

@@ -145,6 +151,10 @@ typedef struct {
145151
extern _PyIO_State *_PyIO_get_module_state(void);
146152
extern PyObject *_PyIO_get_locale_module(_PyIO_State *);
147153

154+
#ifdef MS_WINDOWS
155+
extern char _PyIO_get_console_type(PyObject *);
156+
#endif
157+
148158
extern PyObject *_PyIO_str_close;
149159
extern PyObject *_PyIO_str_closed;
150160
extern PyObject *_PyIO_str_decode;

0 commit comments

Comments
 (0)