Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

Summary (also posted to tracker):

Modules/_datetimemodule.c and Lib/datetime.py do not behave identically. Specifically, the strftime functions do not match when passed a format string
terminated with a '%'. The C function performs an explicit check for this condition, and raises a ValueError on it. The Py version does not perform this check. Both pass the
format string (after doing substitutions for %z, %Z, and %f tags) to the system strftime or wcfstime, depending on platform. These live within the python time module. The
time module wrapper function does not perform this check.

This situation leads to a scenario in which, for example, "%D %" passed to datetime.strftime (with the C extension included) raises a value error. The same string passed to
time.strftime returns "mm/dd/yy %", at least on OSX. Furthermore, if Python is built without the C module, "mm/dd/yy %" is returned when datetime.strftime is called.

To summarise, there are two problems: (1) datetime does not comply with PEP-399, and (2) a higher-order module raises an exception on a case that the (exposed) lower-order
module has no problem with, causing a mismatch in behavior between datetime.strftime and time.strftime.

This PR attempts to fix this problem by removing the case check from the datetime C module. This solves both (1) and (2).

There was much talk on the issue thread about there existing a test case for time.strftime that documented a platform-dependent failure on a dangling '%'. I wish to note
that my patch does not touch the time module at all, it only removes a seemingly unnecessary check in the datetime C module.

@michaelsaah @tirkarthi Can either of you point to the existing test case on this? I cannot find it.

I think that this still needs a new test, because obviously the existing test case does not cover it specifically. Ideally you would want to test that this does not fail for datetime.strftime specifically. If there's a platform-dependent failure in time.strftime, then you can write the test such that the datetime test is skipped if the time test would fail, e.g.:

# Whether datetime.strftime can handle a trailing % is platform-dependent,
# so detect whether we're on one of the platforms where this fails ([bpo-35066](https://bugs.python.org/issue35066))
try:
    time.strftime('%')
    skip_trailing_percent_strftime = False
except ValueError:
    skip_trailing_percent_strftime = True

This is assuming that we don't already know the platforms where the trailing-% fails. If we know which platforms this fails on, then both the time.strftime and datetime.strftime tests should be skipped on the platform with the known failure.

@pganssle In the tracker I was referring to

@tirkarthi Hm. That is a very unfortunately designed test. Even legitimate failures will be swallowed because it treats all ValueErrors as success, even if there is no known platform where this is a failure case.

I think for now we should try the "detect if time.strftime fails, otherwise test datetime.strftime" method, and maybe in the future we can tighten up the failure conditions for time.strftime.

@pganssle Thanks Paul. I'm not sure if the platform failure modes are known; I didn't run into documentation of them during my work. I'll work on adding a test case for datetime that is aware of time.strftime's functionality, but it would help me to know what exactly would happen if stdlib call in time.strftime failed, so that I can catch the failure cleanly. Otherwise I guess I'll be spinning up a bunch of VMs and trying to induce failure.

It looks like there's some error handling being done by _Py_BEGIN_SUPPRESS_IPH, but I'm not sure how this propagates (format_time is a macro for strftime/wcfstime depending on platform):

The macro def'ns are a bit cryptic:

Any help appreciated.

@michaelsaah I don't think you have to worry about any of that. We know that time.strftime('%') should either raise ValueError or succeed. You can just write a test for datetime.strftime that assumes that datetime_instance.strftime('%') returns '%', and then skip that test conditional on the logic I described in my earlier comment (i.e. skip it on all platforms where time.strftime('%') raises ValueError).

@pganssle Ah ok, I was reading the comment in the test-in-question incorrectly. I see the spec now. Thanks.

Just added the test. Will probably open another PR to try to cleanup the datetime.strftime tests, they're all over the place.

Interestingly, looks like there's been some confusion about this before:

@michaelsaah You might want to add a NEWS entry given the impact and it will also help in keeping the PR green : https://devguide.python.org/committing/#what-s-new-and-news-entries

@pganssle I think you'll like the latest change. Behavior w/ regard to null bytes now matches existing code, as far as I can tell.

@michaelsaah Sorry for the days of silence.

I think the behavior of this is now fixed. I'm not sure how I feel about decrementing the pointer, though. I think the two most obviously viable solutions (barring any that dramatically refactor the code, which I have not considered):

1. Checking for a null at the beginning and the end of the loop
2. Checking for a null at the beginning of the loop, but decrementing the pointer (as you've done in this case) when you find a second null

The first one is nice because it doesn't go through unnecessary iterations of the loop, but the second one keeps the null checking logic as close as possible to the point where the pointer is incremented, which keeps the amount of time between reaching the end of the string and knowing we've reached it to a minimum.

I think the only other decent option would be something like this:

while ((ch = *pin++) != '\0') {
    if(ch != '%') {
        ptoappend = pin - 1;
        ntoappend = 1;
    } else if (*(pin + 1) != '\0') {
        if ((ch = *pin++) == 'z') {
            ...

I am increasingly convincing myself that number 2 (your current version) is the right way to go, so I say we go ahead and merge this if no one else objects.

By the way, thank you for your patience with this @michaelsaah, particularly since I know I created some extra work for you with my ill-considered alternative approaches.

Wow we really commented past each other there. No worries about the extra work, you've been a very clear and thorough reviewer, and I'm doing this for fun while looking for a new job anyway.

I originally didn't want to decrement the pointer either, but I'm convinced it's safe, since we only do that if we've already seen a '%', which means we won't be decrementing past the beginning of the buffer.

After staring at this for hours over the past couple weeks, I'm not convinced that there's any way to make the control flow more intuitive while keeping the null-checking contained.

As far as the difference between (1) and (2), I think (2) fits more naturally with the case-checking semantics that are already there, but it's a toss-up. I don't think either is more confusing (or clearer) than the other.

As an aside, I don't see how my solution leads to unnecessary traversals of the loop. As soon as it sees a '%' followed by a '\0', it copies the '%' and breaks on the next while condition check. Do you mean that it breaks on the while condition instead of at the bottom of the loop body?

CC: @abalkin

I think this is ready to be merged.

Thanks @michaelsaah for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

GH-11550 is a backport of this pull request to the 3.7 branch.