Implement job queues for desktop dive #1534
Conversation
BryonLewis
left a comment
There was a problem hiding this comment.
Just some minor suggestions and questions. Feel free to push back on any if you think they are out of scope for this PR.
BryonLewis
left a comment
There was a problem hiding this comment.
Sorry, one more additional change regarding how the awaiting is calculated. It works for bulk import but for single file imports it doesn't update properly.
| watch(recentHistory, () => { | ||
| queuedConversionDatasetIds.value = []; | ||
| cpuJobQueue.jobSpecs.forEach((spec: Job) => { | ||
| if (spec.type === JobType.Conversion) { | ||
| queuedConversionDatasetIds.value.push(spec.meta.id); | ||
| } | ||
| }); | ||
| }); |
There was a problem hiding this comment.
So this works properly with bulk import but if I import multiple data using the Import Video quickly the subsquent data files aren't placed in the awaiting state.
I think this is related to watching recentHistory. RecentHistory is in jobs.ts and is updated when job-update is called. So when you batch load all the data is already in cpuJobQueue and the proper conversionIds are added. If you add videos one at a time only the first job when it starts calls job-update, or you have to wait until it calls job-update again for it to update the list and check the cpuJobQueue.jobSpecs. This may be fixed by simply watching () => cpuJobQueue.jobSpecs instead of RecentHistory because really you want to know that's in the cpuJobQueue of type Conversion at the time it changes. You would probably have to watch deep: true because you're looking at a non-reactive array. There could also be a better way than deeply watching this queue that I don't know of.
There was a problem hiding this comment.
From jobs.ts I'm not exporting queued jobs wrapped in reactive. I've updated Recent.vue to watch this instead. 9111c51
Fix #1532
Prevent batch pipelines/training from hogging resources when running in desktop mode by sequentially running jobs.
Introduces two queues for running jobs in desktop job separated by the resource used (CPU or GPU). Now, instead of invoking the
ipcServicedirectly to start a job, theApifunctions add the job arguments to the appropriate queue. Once a queued job's arguments are up, the IPC call is made by the queue to start the job.This involved some refactoring of the import logic. Previously, the job to convert imported media was started deep within the
finalizeImportlogic. Now,finalizeImportreturns an instance ofConversionArgs. It is the responsibility of consuming code to inspect themediaListand kick off a conversion job as appropriate. Tests were updated to reflect this new paradigm.Job types have also been refactored to include a
typefield, whose value belongs to the newJobTypeenum.The "Jobs" tab now contains two sections: "Job History" - which contains exactly what the previous "Jobs" tab did. It displays the currently running job and previously completed jobs. The "Queued Jobs" tab is new and displays a list of queued jobs.