save cwl.input.json to keep collection if large#83
save cwl.input.json to keep collection if large#83jrandall wants to merge 2 commits intoarvados:mainfrom
Conversation
|
This is a good idea, workbench won't know how to show workflow inputs (but they are very large so that's probably okay). Changing the literal form from "json" to "text" is also likely to cause regressions. |
|
What's the status of this PR? |
|
I think the status is that it potentially breaks the ability to display workflow inputs in Workbench, so in order to accept the fix it would need to be paired with an update to Workbench (or at least some investigation of the side effects of this change on Workbench). |
|
@tetron I've lost track of exactly where this ended up, but it looks like I did implement some fixes on the workbench side to support Also see: https://dev.arvados.org/issues/13685 My recollection is that there are two related performance fixes related to handling of CWL workflows with large numbers of inputs. This PR touches both of them. One is to implement The other fix is to store very large content (defined here as >1MB although that should probably have a config knob) in keep rather than inlining it into the mounts structure (which then gets passed around and repeatedly parsed as part of the container / container_request objects). This further speeds up processing of containers / container requests because they are much smaller. |
If cwl.input.json is larger than 1MB, save it to keep rather than inlining it in the container_request mounts entry.