Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Refactor: Pipeline Executors #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Parent: Develop
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

Refactor: Pipeline Executors #177

wants to merge 9 commits into from

Conversation

@rsteele5
Copy link
Collaborator

Goals:

  • Consolidate pipeline executor functionality (Extract, Classification, and [LLM] Indexing)
    • Reducing code duplication
    • Making pipeline related parsing and execution functions consistent (and modular)
  • Reorganize certain executor into functional packages
    • NOTE: this only effects PipelineExecutor and DocumentLLMPipelineExecutor.
    • PipelineExecutor has become a base class and is not longer intended for direct instantiation.
    • DocumentLLMPipelineExecutor is now LLMIndexerExecutor
    • MetaMergeExecutor has been derived from the merge_metadata function and is no longer in PipelineExecutor
  • Convert all lingering executor imports to absolute imports

Additional Requested Feature:

  • Add a force OCR regeneration feature to the TextExtractionExecutor
    • This executor attempts to parse a "force" attribute from an "ocr" "extract" feature in the request's feature list. If true, any existing or cached OCR will be ignored.
    • Example: {"name":"ocr","type":"extract","force":true}
  • check and update documentation. See guide and ask the team.

rsteele added 9 commits September 11, 2025 16:05
fix: Remove circular import in api _init_.py related to request param…
ce9e2a9 
…eter parsing
chore: remove unused OCRExecutor class
ccf0843
fix: Functionally isolate util scripts to avoid circular imports
1f27174 
* Replace relative imports in executor init's with absolute imports
* Move extract.util to executor package level
* Separate extract.util into asset and into asset_util.py and request_util.py
refactor: Consolidate pipeline related functions
65a8c1e
refactor: move meta_merge_executor from merge to meta package
8ba1c31
refactor: rebase onto executor fixes
9765b30 
* fix imports
refactor: Align the pipelines and add pydocs
582e23e 
* moved store_metadata pipeline function to base_pipeline.py
* All pipelines generate their root_asset_dir from the MARIE_CACHE dir
* Removed classification processing from llm_pipeline.py
* Add OCR disable flag to classification pipeline (default True)
* generalized pipeline component enabled status
feat: Add ocr force request feature parse to text_extraction_executor.py
816156c 
* Feature example:{"name":"ocr","type":"extract","force":true}
refactor: convert lingering executor imports to absolute imports
1d9ef37
@rsteele5 rsteele5 force-pushed the refactor/pipeline-executors branch from 1ae0e05 to 1d9ef37 Compare September 11, 2025 21:18
@rsteele5 rsteele5 mentioned this pull request Sep 23, 2025
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rsteele5