Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@BryonLewis
Copy link
Collaborator

@BryonLewis BryonLewis commented Aug 22, 2025

resolves #1505

  • Update the AssetStore import so it will now support possible annotation file types (json/csv).

Importing Logic:

Below is the logic as each file is imported (GirderItem) in this case:

  1. The current logic when importing will attempt to see if the item is a image-sequence a video or a large image
  2. If it is a video file it will place it in a folder with the same name as the file but minus the extension.
  3. If it is an image sequence it will label the parent as dataset type of image-sequence
  4. If there are annotation files it will first to check and see if parent is of image-sequence and do nothing if it is (the post process will automatically import the file)
  5. If it isn't of a Dataset type it will check the parent folder and attempt to find a folder with the same name as the annotation file minus the extension. If found it will then move the item into that folder.
  6. If it can't find anything (this means it was processed before the video file) it will mark it as AnnotationFileFutureProcess so that once everything is imported it can move these files around.

Once all Items are imported a secondary process begins to kick off transcoding jobs. Injected into that is a new process to manage all dangling AnnotationFileFutureProcess files to place them in the proper folder.

  1. There is a global check for all items that have AnnotationFileFutureProcessMarker set to True
  2. For each annotation file it will check if the parent-folder is a image-sequence or large-image and will just remove the the AnnotationFileFutureProcessMarker from the metadata on the item.
  3. If not it will then look for a folder with the same base_name (annotation filename without extension) inside the parent folder and place the annotation file inside that folder.
  4. It loops through all folders that were imported in the system to do this check.

Finally it kicks off process to import all video files as datasets by running the quick check to see if files need to be transcoded. During this process it calles the /postprocess endpoint that will check and look for CSV/JSON files in the same folder as the DIVEDataset and will import it as annotations if it exists.

Instructions for Testing are found in the ./samples/scripts/assetStoreImport/README.md file

Discussion of Technical Implementation:

DIVE-AssetStoreImport-TechnicalDetails_small.mp4

Testing Process:

DIVE-AssetStoreImport-Demo_small.mp4

@BryonLewis BryonLewis marked this pull request as ready for review September 10, 2025 15:02
naglepuff
naglepuff previously approved these changes Sep 10, 2025
Copy link
Collaborator

@naglepuff naglepuff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! From what I can tell this all works as intended. I found the test script to be very helpful and easy to use.

When I was looking at some of the scripts in VSCode the editor was complaining about a blank line missing.

Comment on lines 94 to 96
minio_ip = get_container_ip(minio_container, "dive_default")
# Helper function: run mc inside persistent container
def mc_cmd(*args, capture_output=False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a huge deal, but I think there should be a blank line somewhere in here. My editor was complaining

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup my linter was yelling about this as well

@BryonLewis BryonLewis merged commit 7b70079 into main Sep 11, 2025
4 checks passed
@BryonLewis BryonLewis deleted the assetstore-import-updates branch September 11, 2025 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AssetStore Bucket Import Annotation Support

2 participants