Oooooo, this is awesome! I think this is exactly what some people have been asking for.

I do think you're right though, as it stands now, looping through all the records, may be ultimately inefficient for large Jobs. Thankfully, I think we could lean on the Django/Mongo ORM to pull these counts pretty quickly.

Don't have a one liner handy, but something like the following might work:

# get Job's records as QuerySet (MongoEngine, but very similar to native Django SQL ORM)
job_records = Job.objects.get(pk=224).get_records()

# get OAI sets from Job records
job_records.values_list('oai_set').distinct('oai_set')
Out[21]: 
['wayne:collectioncfai',
 'wayne:collectionhermanmiller',
 'wayne:collectionrencen',
 'wayne:collectionmim']

job_records.filter(oai_set='wayne:collectioncfai').count()
Out[22]: 2292

Then, to avoid calculating these each time a Job is loaded, one option may be to store in a Job's job_details, which is a JSON object that is storing these exact kinds of things (field mapping metrics, etc.). Could be stored on Job finish, or first time Job is loaded.

But it's looking awesome. Happy to keep spitballing, but in short, I would think leaning on the ORM might be a good option. And if necessary, it that ends up being costly for huge jobs, could count with Spark and write to job_details that way.

Uh oh!

include OAI harvest details in Job details #374

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions