Thanks to visit codestin.com
Credit goes to github.com

Skip to content

S3 assetstore import job #1525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BryonLewis merged 3 commits into from
Sep 18, 2025
Merged

S3 assetstore import job #1525

BryonLewis merged 3 commits into from
Sep 18, 2025

Conversation

@BryonLewis
Copy link
Collaborator

@BryonLewis BryonLewis commented Sep 18, 2025 •
edited
Loading

This updates the importing of S3 Assets to create a master job that spawns other jobs for the conversion.
Before this if you had 1000 videos it would immediately spawn 1000 jobs. This spawns and processes jobs one at a time to make the job management a bit easier. If you cancel the main job it will stop adding more jobs.

  • Update crud_rpc.postprocess to return the job_ids of any jobs that are spawned
    • Updated the client side to handle this new format as well
  • Added a dive_batch_postprocess task that will search through a folder and continually spawn subsequent jobs for all data that is 'MarkedForPostProcess'.
    • As a job completes the next job will start
    • If there are errors it will log it and continue the next job
    • If a user cancels the main job it will stop adding new jobs
  • Updated the event.py for asset store importing so it will kick off this main job instead of thousands of smaller jobs.
DIVE-20250918-S3BatchImport_small.mp4

@BryonLewis BryonLewis marked this pull request as ready for review September 18, 2025 16:58
naglepuff
naglepuff reviewed Sep 18, 2025
View reviewed changes
naglepuff
naglepuff approved these changes Sep 18, 2025
View reviewed changes
Copy link
Collaborator

@naglepuff naglepuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Only kind of clunky thing is that if a user wants to go to the main job via the job list, it could be difficult depending on how fast the child jobs are spawning. If they spawn to fast, it could be hard to click into the main job. This is alleviated by the fact that starting the import shows a notification that the user can click to go right to the main job.

@BryonLewis
Copy link
Collaborator Author

BryonLewis commented Sep 18, 2025

This looks good to me. Only kind of clunky thing is that if a user wants to go to the main job via the job list, it could be difficult depending on how fast the child jobs are spawning. If they spawn to fast, it could be hard to click into the main job. This is alleviated by the fact that starting the import shows a notification that the user can click to go right to the main job.

This was something David and I were talking about the other day, the same thing happens in Slicer CLI batch jobs. You can also use the type filter to quickly filter for type "DIVE Batch Process Import" and it will only show those jobs. This would only be used by an admin of the system though, so if it is a little cumbersome I think that's okay.

@BryonLewis BryonLewis merged commit 888e8b5 into main Sep 18, 2025
4 checks passed
@BryonLewis BryonLewis deleted the s3-assetstore-import-job branch September 18, 2025 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@naglepuff naglepuff naglepuff approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@BryonLewis @naglepuff