Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-98836: Extend PyUnicode_FromFormat() #98838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
May 21, 2023
Merged

gh-98836: Extend PyUnicode_FromFormat() #98838

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
aca2393
gh-98836: Extend PyUnicode_FromFormat()
serhiy-storchaka Aug 10, 2022
4fdf596
Fix docs formatting.
serhiy-storchaka Oct 29, 2022
13d4c97
Add tests for integer conversions.
serhiy-storchaka Oct 30, 2022
e69d8b0
Merge branch 'main' into capi-unicode-fromformat
serhiy-storchaka Nov 3, 2022
3bd9d3e
Add more tests.
serhiy-storchaka Nov 3, 2022
1e6094b
Finish documentation.
serhiy-storchaka Nov 3, 2022
8942ca3
Merge branch 'main' into capi-unicode-fromformat
serhiy-storchaka Nov 6, 2022
8481bda
Remove duplicated paragraph.
serhiy-storchaka Nov 6, 2022
8885850
Address review comments.
serhiy-storchaka Nov 6, 2022
9ffdb86
Fix syntax error in a comment.
serhiy-storchaka Nov 6, 2022
9ca3764
Fix and refactor tests.
serhiy-storchaka Nov 7, 2022
79a523c
Merge branch 'main' into capi-unicode-fromformat
serhiy-storchaka Nov 20, 2022
91d8393
Add more tests for %ls and %lV.
serhiy-storchaka Nov 20, 2022
9f2f5f2
Merge branch 'main' into capi-unicode-fromformat
serhiy-storchaka May 21, 2023
b86d92d
Try to silence Sphinx warnings.
serhiy-storchaka May 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
gh-98836: Extend PyUnicode_FromFormat() 
* Support for conversion specifiers o (octal) and X (uppercase hexadecimal).
* Support for length modifiers j (intmax_t) and t (ptrdiff_t).
* Length modifiers are now applied to all integer conversions.
* Support for wchar_t C strings (%ls and %lV).
* Support for variable width and precision (*).
* Support for flag - (left alignment).
  • Loading branch information
@serhiy-storchaka
serhiy-storchaka committed Oct 29, 2022
commit aca2393168ef45535e4245b6852db600dd57a288
228 changes: 143 additions & 85 deletions Doc/c-api/unicode.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,98 +394,149 @@ APIs:
arguments, calculate the size of the resulting Python Unicode string and return
a string with the values formatted into it. The variable arguments must be C
types and must correspond exactly to the format characters in the *format*
ASCII-encoded string. The following format characters are allowed:

.. % This should be exactly the same as the table in PyErr_Format.

.. tabularcolumns:: |l|l|L|

+-------------------+---------------------+----------------------------------+
| Format Characters | Type | Comment |
+===================+=====================+==================================+
| :attr:`%%` | *n/a* | The literal % character. |
+-------------------+---------------------+----------------------------------+
| :attr:`%c` | int | A single character, |
| | | represented as a C int. |
+-------------------+---------------------+----------------------------------+
| :attr:`%d` | int | Equivalent to |
| | | ``printf("%d")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%u` | unsigned int | Equivalent to |
| | | ``printf("%u")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%ld` | long | Equivalent to |
| | | ``printf("%ld")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%li` | long | Equivalent to |
| | | ``printf("%li")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%lu` | unsigned long | Equivalent to |
| | | ``printf("%lu")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%lld` | long long | Equivalent to |
| | | ``printf("%lld")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%lli` | long long | Equivalent to |
| | | ``printf("%lli")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%llu` | unsigned long long | Equivalent to |
| | | ``printf("%llu")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%zd` | :c:type:`\ | Equivalent to |
| | Py_ssize_t` | ``printf("%zd")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%zi` | :c:type:`\ | Equivalent to |
| | Py_ssize_t` | ``printf("%zi")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%zu` | size_t | Equivalent to |
| | | ``printf("%zu")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%i` | int | Equivalent to |
| | | ``printf("%i")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%x` | int | Equivalent to |
| | | ``printf("%x")``. [1]_ |
+-------------------+---------------------+----------------------------------+
| :attr:`%s` | const char\* | A null-terminated C character |
| | | array. |
+-------------------+---------------------+----------------------------------+
| :attr:`%p` | const void\* | The hex representation of a C |
| | | pointer. Mostly equivalent to |
| | | ``printf("%p")`` except that |
| | | it is guaranteed to start with |
| | | the literal ``0x`` regardless |
| | | of what the platform's |
| | | ``printf`` yields. |
+-------------------+---------------------+----------------------------------+
| :attr:`%A` | PyObject\* | The result of calling |
| | | :func:`ascii`. |
+-------------------+---------------------+----------------------------------+
| :attr:`%U` | PyObject\* | A Unicode object. |
+-------------------+---------------------+----------------------------------+
| :attr:`%V` | PyObject\*, | A Unicode object (which may be |
| | const char\* | ``NULL``) and a null-terminated |
| | | C character array as a second |
| | | parameter (which will be used, |
| | | if the first parameter is |
| | | ``NULL``). |
+-------------------+---------------------+----------------------------------+
| :attr:`%S` | PyObject\* | The result of calling |
| | | :c:func:`PyObject_Str`. |
+-------------------+---------------------+----------------------------------+
| :attr:`%R` | PyObject\* | The result of calling |
| | | :c:func:`PyObject_Repr`. |
+-------------------+---------------------+----------------------------------+
ASCII-encoded string.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add something like [[fill]align][sign][z][#][0][width][grouping_option][.precision][type]? Example taken from: https://docs.python.org/dev/library/string.html#format-specification-mini-language

The printf format is complex, and it's not obvious in which order each part should be written. I understand that it's something like:

%[conversion flags][minimum width][.precision][length modifier][conversion type]

Or if you prefer a shorter version:

%[conversion][width][.precision][modifier][conversion]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied the following paragraphs from the description of the print-like formatting in Python: https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting.

I think that it also can benefit from from similar changes, but let leave it to other issue.

A conversion specifier contains two or more characters and has the following
components, which must occur in this order:

#. The ``'%'`` character, which marks the start of the specifier.

#. Conversion flags (optional), which affect the result of some conversion
types.

#. Minimum field width (optional).
If specified as an ``'*'`` (asterisk), the actual width is given in the
next argument, which must be of type int, and the object to convert comes
after the minimum field width and optional precision.

#. Precision (optional), given as a ``'.'`` (dot) followed by the precision.
If specified as ``'*'`` (an asterisk), the actual precision is given in
the next argument, which must be of type :c:type:`int`, and the value to
convert comes after the precision.

#. Length modifier (optional).

#. Conversion type.

The conversion flag characters are:

.. tabularcolumns:: |l|L|

+-------+-------------------------------------------------------------+
| Flag | Meaning |
+=======+=============================================================+
| ``0`` | The conversion will be zero padded for numeric values. |
+-------+-------------------------------------------------------------+
| ``-`` | The converted value is left adjusted (overrides the ``0`` |
| | flag if both are given). |
+-------+-------------------------------------------------------------+

The length modifiers for following integer conversions (``d``, ``i``,
``o``, ``u``, ``x``, or ``X``) specify the type of the argument
(:c:type:`int` by default):

.. tabularcolumns:: |l|L|

+----------+-----------------------------------------------------+
| Modifier | Types |
+==========+=====================================================+
| ``l`` | :c:type:`long` or :c:type:`unsigned long` |
+----------+-----------------------------------------------------+
| ``ll`` | :c:type:`long long` or :c:type:`unsigned long long` |
+----------+-----------------------------------------------------+
| ``j`` | :c:type:`intmax_t` or :c:type:`uintmax_t` |
+----------+-----------------------------------------------------+
| ``z`` | :c:type:`size_t` or :c:type:`ssize_t` |
+----------+-----------------------------------------------------+
| ``t`` | :c:type:`ptrdiff_t` |
+----------+-----------------------------------------------------+

The length modifier ``l`` for following conversions ``s`` or ``V`` specify
that the type of the argument is :c:expr:``const wchar_t*``.

The conversion specifiers are:

.. list-table::
:widths: auto
:header-rows: 1

* - Conversion Specifier
- Type
- Comment

* - ``%``
- *n/a*
- The literal ``%`` character.

* - ``d``, ``i``
- Specified by the length modifier
- The decimal representation of a signed C integer.

* - ``u``
- Specified by the length modifier
- The decimal representation of an unsigned C integer.

* - ``o``
- Specified by the length modifier
- The octal representation of an unsigned C integer.

* - ``x``
- Specified by the length modifier
- The heximal representation of an unsigned C integer (lowercase).

* - ``X``
- Specified by the length modifier
- The heximal representation of an unsigned C integer (uppercase).

* - ``c``
- :c:type:`int`
- A single character.

* - ``s``
- :c:expr:`const char*` or :c:expr:`const wchar_t*`
- A null-terminated C character array.

* - ``p``
- :c:expr:`const void*`
- The hex representation of a C pointer.
Mostly equivalent to ``printf("%p")`` except that it is guaranteed to
start with the literal ``0x`` regardless of what the platform's
``printf`` yields.

* - ``A``
- :c:expr:`PyObject*`
- The result of calling :func:`ascii`.

* - ``U``
- :c:expr:`PyObject*`
- A Unicode object.

* - ``V``
- :c:expr:`PyObject*`, :c:expr:`const char*` or :c:expr:`const wchar_t*`
- A Unicode object (which may be ``NULL``) and a null-terminated
C character array as a second parameter (which will be used,
if the first parameter is ``NULL``).

* - ``S``
- :c:expr:`PyObject*`
- The result of calling :c:func:`PyObject_Str`.

* - ``R``
- :c:expr:`PyObject*`
- The result of calling :c:func:`PyObject_Repr`.

.. note::
The width formatter unit is number of characters rather than bytes.
The precision formatter unit is number of bytes for ``"%s"`` and
The precision formatter unit is number of bytes or :c:type:`wchar_t`
items (if the length modifier ``l`` is used) for ``"%s"`` and
``"%V"`` (if the ``PyObject*`` argument is ``NULL``), and a number of
characters for ``"%A"``, ``"%U"``, ``"%S"``, ``"%R"`` and ``"%V"``
(if the ``PyObject*`` argument is not ``NULL``).

.. [1] For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
zu, i, x): the 0-conversion flag has effect even when a precision is given.
.. note::
Unlike to C :c:func:`printf` the ``0`` flag has effect even when
a precision is given for integer conversions (``d``, ``i``, ``u``, ``o``,
``x``, or ``X``).

.. versionchanged:: 3.2
Support for ``"%lld"`` and ``"%llu"`` added.
Expand All @@ -498,6 +549,13 @@ APIs:
``"%V"``, ``"%S"``, ``"%R"`` added.

.. versionchanged:: 3.12
Support for conversion specifiers ``o`` and ``X``.
Support for length modifiers ``j`` and ``t``.
Length modifiers are now applied to all integer conversions.
Length modifier ``l`` is now applied to conversion specifiers ``s`` and ``V``.
Support for variable width and precision ``*``.
Support for flag ``-``.

An unrecognized format character now sets a :exc:`SystemError`.
In previous versions it caused all the rest of the format string to be
copied as-is to the result string, and any extra arguments discarded.
Expand Down
3 changes: 2 additions & 1 deletion Lib/test/test_unicode.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2880,10 +2880,11 @@ def check_format(expected, format, *args):
# check for crashes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment is misleading. Python doesn't crash but reject invalid format string with a SyntaxError. Would you mind to rephrase the comment? Like: # test invalid format strings?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Restored the part of the older comment.

for fmt in (b'%', b'%0', b'%01', b'%.', b'%.1',
b'%0%s', b'%1%s', b'%.%s', b'%.1%s', b'%1abc',
b'%l', b'%ll', b'%z', b'%ls', b'%lls', b'%zs'):
b'%l', b'%ll', b'%z', b'%lls', b'%zs'):
with self.subTest(fmt=fmt):
self.assertRaisesRegex(SystemError, 'invalid format string',
PyUnicode_FromFormat, fmt, b'abc')

self.assertRaisesRegex(SystemError, 'invalid format string',
PyUnicode_FromFormat, b'%+i', c_int(10))

Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/C API/2022-10-29-10-13-20.gh-issue-98836.Cy5h_z.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Add support of more formatting options (left aligning, octals, uppercase
hexadecimals, :c:type:`intmax_t`, :c:type:`ptrdiff_t`, :c:type:`wchar_t` C
strings, variable width and precision) in :c:func:`PyUnicode_FromFormat` and
:c:func:`PyExc_Format`.
5 changes: 1 addition & 4 deletions Modules/_ssl.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1326,10 +1326,8 @@ _get_peer_alt_names (_sslmodulestate *state, X509 *certificate) {
p[0], p[1], p[2], p[3]
);
} else if (name->d.ip->length == 16) {
/* PyUnicode_FromFormat() does not support %X */
unsigned char *p = name->d.ip->data;
len = sprintf(
buf,
v = PyUnicode_FromFormat(
"%X:%X:%X:%X:%X:%X:%X:%X",
p[0] << 8 | p[1],
p[2] << 8 | p[3],
Expand All @@ -1340,7 +1338,6 @@ _get_peer_alt_names (_sslmodulestate *state, X509 *certificate) {
p[12] << 8 | p[13],
p[14] << 8 | p[15]
);
v = PyUnicode_FromStringAndSize(buf, len);
} else {
v = PyUnicode_FromString("<invalid>");
}
Expand Down
8 changes: 4 additions & 4 deletions Modules/_testcapi/unicode.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,10 +358,10 @@ test_string_from_format(PyObject *self, PyObject *Py_UNUSED(ignored))
CHECK_FORMAT_1( "%c", "c", 'c');
CHECK_FORMAT_1( "%0c", "c", 'c');
CHECK_FORMAT_1("%00c", "c", 'c');
CHECK_FORMAT_1( "%2c", "c", 'c');
CHECK_FORMAT_1("%02c", "c", 'c');
CHECK_FORMAT_1("%.0c", "c", 'c');
CHECK_FORMAT_1("%.2c", "c", 'c');
// CHECK_FORMAT_1( "%2c", "c", 'c');
// CHECK_FORMAT_1("%02c", "c", 'c');
// CHECK_FORMAT_1("%.0c", "c", 'c');
// CHECK_FORMAT_1("%.2c", "c", 'c');

// Integers
CHECK_FORMAT_1("%d", "123", (int)123);
Expand Down
5 changes: 1 addition & 4 deletions Modules/selectmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1848,14 +1848,11 @@ static PyObject *

kqueue_event_repr(kqueue_event_Object *s)
{
char buf[1024];
PyOS_snprintf(
buf, sizeof(buf),
return PyUnicode_FromFormat(
"<select.kevent ident=%zu filter=%d flags=0x%x fflags=0x%x "
"data=0x%llx udata=%p>",
(size_t)(s->e.ident), (int)s->e.filter, (unsigned int)s->e.flags,
(unsigned int)s->e.fflags, (long long)(s->e.data), (void *)s->e.udata);
return PyUnicode_FromString(buf);
}

static int
Expand Down
8 changes: 2 additions & 6 deletions Modules/socketmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1286,8 +1286,6 @@ setbdaddr(const char *name, bdaddr_t *bdaddr)
static PyObject *
makebdaddr(bdaddr_t *bdaddr)
{
char buf[(6 * 2) + 5 + 1];

#ifdef MS_WINDOWS
int i;
unsigned int octets[6];
Expand All @@ -1296,16 +1294,14 @@ makebdaddr(bdaddr_t *bdaddr)
octets[i] = ((*bdaddr) >> (8 * i)) & 0xFF;
}

sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X",
return PyUnicode_FromFormat("%02X:%02X:%02X:%02X:%02X:%02X",
octets[5], octets[4], octets[3],
octets[2], octets[1], octets[0]);
#else
sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X",
return PyUnicode_FromFormat("%02X:%02X:%02X:%02X:%02X:%02X",
bdaddr->b[5], bdaddr->b[4], bdaddr->b[3],
bdaddr->b[2], bdaddr->b[1], bdaddr->b[0]);
#endif

return PyUnicode_FromString(buf);
}
#endif

Expand Down
Loading