Generic support for fileGrp whitelist/blacklist #1139
Conversation
d81543f to
2c1faec
Compare
MehmedGIT
left a comment
There was a problem hiding this comment.
Looks good, I have not tested using the CLI, but the tests already cover that.
Badly copy/pasted, hooray for code reviews Co-authored-by: Mehmed Mustafa <[email protected]>
|
@kba IIUC this does not fix #506, because it merely restricts what is downloaded/zipped, not what is actually kept in the METS (as originally intended). The former also has its use-cases, so this might still be useful, but let's look at the original use-cases again:
You would still see the full refs in the METS, despite the filter. If these are URLs, then they would still be fully downloaded (at runtime) by our processors. Otherwise (i.e. if they are merely local paths), the missing refs will be useless, because there is no way of reconstructing their file content. (So why keep them in the METS anyway?) |
There are use cases, where it might be beneficial to only operate on a subset of the file groups of an OCR-D workspace
This PR introduces an inclusion/exclusion mechanism to filter which fileGrps should be allowed to
OcrdMets.find_files.Example for cloning
Download for the pages in range
PHYS_0010..PHYS0019all files, except those in thePRESENTATIONfile groupExample: workspace bagger
E.g. to only include the
DEFAULTandTESSfileGrps:Or to include all fileGrps except
PRESENTATION(which has thegoobiURLs in our collections):Programmatically
Programmatically, the relevant kwargs are
include_fileGrpandexclude_fileGrp.Related
fixes #356
related #582
fixes #506
fixes #383