Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

It probably would be better to raise an attributeError instead of a valueError here since you are trying to access an attribute a closed zipfile doesn’t have

This behavior simply resembles open() and write(), which raises a ValueError in various cases. Furthermore there has been a change from raising RuntimeError since Python 3.6:

Changed in version 3.6: Calling open() on a closed ZipFile will raise a ValueError. Previously, a RuntimeError was raised.

Changed in version 3.6: Calling write() on a ZipFile created with mode 'r' or a closed ZipFile will raise a ValueError. Previously, a RuntimeError was raised.

Nicely inform @ubershmekel, @barneygale, @merwok, and @wimglenn about this PR. This should be more desirable and flexible than the previous PR, although cares must be taken as there might be a potential risk on the algorithm about reclaiming spaces.

The previous PR is kept open in case some folks are interested in it. Will close when either one is accepted.

@emmatyping Still pending review... I'd like to know if there is still any problem about this PR?

Unfortunately it's premature to move forward with this PR until the discussion in the issue is resolved. There are open questions about whether we should add this API or not, and if we do what it would look like. That needs to be worked out in the issue before this PR can move forward, sorry!

Quickly released as zipremove PyPI package that supports Python >= 3.9.

I'm tired of doing all the migration jobs and dealing with all the compatibility crabs over and over again. This work will be migrated to that package and updating of this branch will be halted, until folks are sincerely going to merge into the standard library.

And an additional question: whether this should be included in the current PR, or proposed separately as a follow-up?

This sounds like another feature, but it does sound useful. Perhaps it should have its own issue (if it doesn't already). I'd say let's limit the scope here to addressing the linked issue.

Should strict_descriptor option for repack() default to True or False?

I don't love that there's two implementations here. My reading from the code is that unsigned descriptors are deprecated, so maybe it would be enough to simply omit support for them, opt for the faster implementation, and (maybe) warn if such a descriptor is encountered that it's unsupported. How prevalent are such descriptors?

Should strict_descriptor option for repack() default to True or False?

I don't love that there's two implementations here. My reading from the code is that unsigned descriptors are deprecated, so maybe it would be enough to simply omit support for them, opt for the faster implementation, and (maybe) warn if such a descriptor is encountered that it's unsupported. How prevalent are such descriptors?

Actually there are three implementations. 😂

Therefore the brief algorithm/condition for the slow scan is:

1. removed is not provided.
2. Has a local file header like byte sequence (with PK\x03\x04 signature, flag bit 8, and CRC == compress_size == file_size == 0), which should be very unlikely to happen on random bytes.
3. A valid signed data descriptor is not seen.
4. A supported compression method (which can reliably detect stream ending) is not used. (STORED/LZMA) (not sure if LZMA decompressor can be reworked to support it)
5. strict_descriptor==False

It should be quite unlikely to happen even if strict_descriptor==False, but the problem is still that it may be catastrophically slow once it happens, and could be used intentionally and offensively.

The prevalence of the deprecated unsigned data descriptor is hard to tell. Most apps like WinZip, WinRAR, 7z would probably write no data descriptor since they are generally not used in a streaming condition. For streaming cases I think most developers would simply use the ZIP implementation at their hand, and to answer this question we'd have to check the ZIP implementation of popular programming languages like C, Java, JS, php, C#, etc.

Since this feature has been already implemented, I'd prefer to keep it unless there's a strong reason that it should not exist. Defaulting strict_descriptor to False should be enough if it's not welcome.

A minor reason is that we still cannot clearly say that unsigned data descriptor is not supported even if the slow scan is removed, due to the decompression approach. But maybe another choice is that the decompression approach should also be skipped when strict_descriptor==True.

Reworked strict_descriptor=True to ignore any scan of unsigned data descriptors. (danny0838/zipremove@1a07dff)

@danny0838 in reply to #51067 (comment) (sorry about fragmenting the discussion) -

I'm generally supportive of the iterative approach of start simple and expand from that.

Having said that, since you already implemented _remove_members, I don't see why not expose it through a public API, given that ZipFile has other examples for functions that receive a list of members (members arg of extractall) - can simply follow the same pattern for consistency.
I do agree it's much more useful - my own local implementation only removes a single member and I do see the performance benefits of having an ability to remove multiple members, as many times I myself need to remove multiple members in many different files and it would be much beneficial.

I'm not fond of the idea of a 2 step removal (remove/repack), since it can leave the file in state that some may see (me included) as corruption - zip files aren't support to have a CD entry without a corresponding LF entry.
I'd rather take an approach that limits certain things (cannot remove a file from a self extracting archive for example),
and I don't see the large benefit of delayed repacking flexibility, seems like rather an edge case (but I might be wrong here).

Also some of the questions (how to handle a folder) are relevant regardless of whether we take the 1 step or 2 step approach.
If the 2 step approach has more support here, I'll take it - we can always improve it in the future, but I'd prefer to keep things simpler when possible.

And last but not least - perfect is the enemy of good - it's not the last python version, and we can always improve as long as the initial version is reasonable, which I believe it is.

@distantvale

Having said that, since you already implemented _remove_members, I don't see why not expose it through a public API, given that ZipFile has other examples for functions that receive a list of members (members arg of extractall) - can simply follow the same pattern for consistency.

I don't think extractall is a good analogy since it's read-only. Instead you should check for open('w'), write, write_str, and mkdir, each of which is a simple single entry handler.

There are stil design considerations even for your proposed members, for example:

1. Should there be remove plus removeall, or a remove that accepts either a single entry or a list?
2. What to do if identical names are passed? Should they be treated as a set and deduplicated, or be removed for multiple times?
3. What to do if any passed value causes an error (such as non-exist)?

The current implementation of _remove_members actually defers such decisions. This is fine for a private helper method but is probably premature for a public API.

I'm not fond of the idea of a 2 step removal (remove/repack), since it can leave the file in state that some may see (me included) as corruption - zip files aren't support to have a CD entry without a corresponding LF entry.

I don't get this. remove leaves a LF entry without a corresponding CD entry, which is clearly allowed by the ZIP spec and won't be complained by any ZIP app.

I'd rather take an approach that limits certain things (cannot remove a file from a self extracting archive for example), and I don't see the large benefit of delayed repacking flexibility, seems like rather an edge case (but I might be wrong here).

A delayed repacking isn't necessarily done after the archive is closed, it can also happen simply after the method is called and returned.

For example, a repacking after a mixed operations of writing, removing, copying, renaming, and just before archive closing, such as an interactive ZIP file editing tool would do.

It's probably challenging enough to design a one-time remove or removeall to support all such cases.

Also some of the questions (how to handle a folder) are relevant regardless of whether we take the 1 step or 2 step approach.

I don't think there's too much need to dig into such details for low level APIs—just let them work simply like other existing methods. It's not the case for a one-time high level remove—and maybe copy and rename—though.

@danny0838 I can share my own opinions about the desired behavior but of course not everyone would agree:

I don't think extractall is a good analogy since it's read-only. Instead you should check for open('w'), write, write_str, and mkdir, each of which is a simple single entry handler.

I don't see the need to separate into read/write, as I believe they should all be consistent.
If certain functions, such as extract, have 2 versions - for single/multiple files, then it won't be an exception in the API, and if we find it useful, have both.

There are stil design considerations even for your proposed members, for example:

1. Should there be remove plus removeall, or a remove that accepts either a single entry or a list?

I'd follow extract in this case.

2. What to do if identical names are passed? Should they be treated as a set and deduplicated, or be removed for multiple times?

I might be wrong here, but I think add allows adding multiple files with the same name, so in that case I'd remove multiple times. If I'mt wrong about add, I'd treat it as a set.

3. What to do if any passed value causes an error (such as non-exist)?

Just like in extractall, the names must be a subset of namelist().

The current implementation of _remove_members actually defers such decisions. This is fine for a private helper method but is probably premature for a public API.

I don't get this. remove makes a LF entry without a corresponding CD entry, which is clearly allowed by the ZIP spec and won't be complained by any ZIP app.

Ok, fair enough - it just seems less desireable/clean to me, but I might be a minority here.

A delayed repacking isn't necessarily done after the archive is closed, it can also happen simply after the method is called and returned.

For example, a repacking after a mixed operations of writing, removing, copying, renaming, and just before archive closing, such as an interactive ZIP file editing tool would do.

Yes of course, but it might also be called after the file is closed. I can see some performance benefits, but it would seem like something that a low level API would provide, as opposed to a high level one.
But again, I won't die on that hill, if most people find it useful, so be it - either way it's a good way to start from my end.

It's probably challenging enough to design a one-time remove or removeall to support all such cases.

Also some of the questions (how to handle a folder) are relevant regardless of whether we take the 1 step or 2 step approach.

I don't think there's too much need to dig into such details for low level APIs—just let them work simply like other existing methods. It's not the case for a one-time high level remove—and maybe copy and rename—though.

I'm fine with a "naive" API as well, documentation is enough for these cases at this point IMO.

@distantvale

@danny0838 I can share my own opinions about the desired behavior but of course not everyone would agree:

I don't think extractall is a good analogy since it's read-only. Instead you should check for open('w'), write, write_str, and mkdir, each of which is a simple single entry handler.

I don't see the need to separate into read/write, as I believe they should all be consistent. If certain functions, such as extract, have 2 versions - for single/multiple files, then it won't be an exception in the API, and if we find it useful, have both.

The truth is that they are not consistent. When you say they should be consistent, do you mean there should be multiple file version for open('w'), write, write_str, and mkdir, or there should be no multiple file version for extractall, or every of them should be a single method that supports both single and list input?

There are stil design considerations even for your proposed members, for example:

1. Should there be remove plus removeall, or a remove that accepts either a single entry or a list?

I'd follow extract in this case.

2. What to do if identical names are passed? Should they be treated as a set and deduplicated, or be removed for multiple times?

I might be wrong here, but I think add allows adding multiple files with the same name, so in that case I'd remove multiple times. If I'mt wrong about add, I'd treat it as a set.

Actually there is no ZipFile.add. Do you mean write and others?

3. What to do if any passed value causes an error (such as non-exist)?

Just like in extractall, the names must be a subset of namelist().

In extractall each input is handled one by one, and any error causes subsequent inputs not handled. However the current _remove_members handles removing and repacking simultaneously. If any error happens in the middle, the whole repacking is left partially done and the archive will be in an inconsistent state, which is unlikely an acceptable consequence.

Likewise, for extractall providing duplicated names just extract the same entry to the same filesystem path again; for removing this would be a totally different story.

Anyway, the current _remove_members implementation DOESN'T do what you state above. If you favor that approach and really want such behavior, you'd have to work on that PR, doing more tests and feedback (I'm probably not going to keep working on that unless it's the final decision of the issue). It would also be nice if you can provide the code of the implementation you've been using.

@distantvale

@danny0838 I can share my own opinions about the desired behavior but of course not everyone would agree:

I don't think extractall is a good analogy since it's read-only. Instead you should check for open('w'), write, write_str, and mkdir, each of which is a simple single entry handler.

I don't see the need to separate into read/write, as I believe they should all be consistent. If certain functions, such as extract, have 2 versions - for single/multiple files, then it won't be an exception in the API, and if we find it useful, have both.

The truth is that they are not consistent. When you say they should be consistent, do you mean there should be multiple file version for open('w'), write, write_str, and mkdir, or there should be no multiple file version for extractall, or every of them should be a single method that supports both single and list input?

What I'm saying is that there's already precendence in the package for a function that has 2 versions, so it's not out of the ordinary to add another one.

There are stil design considerations even for your proposed members, for example:

1. Should there be remove plus removeall, or a remove that accepts either a single entry or a list?

I'd follow extract in this case.

2. What to do if identical names are passed? Should they be treated as a set and deduplicated, or be removed for multiple times?

I might be wrong here, but I think add allows adding multiple files with the same name, so in that case I'd remove multiple times. If I'mt wrong about add, I'd treat it as a set.

Actually there is no ZipFile.add. Do you mean write and others?

Yup, I mean write.

3. What to do if any passed value causes an error (such as non-exist)?

Just like in extractall, the names must be a subset of namelist().

In extractall each input is handled one by one, and any error causes subsequent inputs not handled. However the current _remove_members handles removing and repacking simultaneously. If any error happens in the middle, the whole repacking is left partially done and the archive will be in an inconsistent state, which is unlikely an acceptable consequence.

The behavior of extractall might not be aligned with the documentation that states that members must be a subset of namelist() - hard to say what the intention was, but I think it's a legitimate check.
I agree re. the inconsistent state, and maybe that's the most significant reason to keep the external API simple for now. Once the initial version is release, expanding it to more functionality is easier.

Likewise, for extractall providing duplicated names just extract the same entry to the same filesystem path again; for removing this would be a totally different story.

Yeah I agree, and since multiple files with the same names are not very common, I'd opt for treating members as a set.

Anyway, the current _remove_members implementation DOESN'T do what you state above. If you favor that approach and really want such behavior, you'd have to work on that PR, doing more tests and feedback (I'm probably not going to keep working on that unless it's the final decision of the issue). It would also be nice if you can provide the code of the implementation you've been using.

My implementation is based on the original PR, so not much there in regards to these options.

In any case I'd proceed with your approach for the time being.