Thanks @gadfort. It sounds like there is a lot of caching going on in the background. Say that I have two independent flows, that take the same input, and a third generating the input. Can they access the same cache? Or is the cache accessible only by the nodes inside the flow?

Got it. Thank you very much.

@gadfort, I have a follow-up question. I'm trying to setup a hierarchical flow. In thinking to have the user create the libraries, specifying the hierarchies and dependencies by composing multiple siliconcompiler.Library instances. Then feed the "top" library into a flow. I want the flow to work over the Library schema and generate nodes that execute operations per library basis, that respect the dependencies captured in the "top" library. Basically generate a flowgraph based on the hierarchical library.

As far as I can see the Library schema doesn't track any dependencies. Is something like this achievable with the builtin functionality? Or do I need to add certain fields for each library?

Thanks!

We do something similar in ZeroSOC: https://github.com/siliconcompiler/zerosoc/blob/00b201ca0d799ec5a432d901169823cec0f5daea/make.py#L205

The flow there is build the core element and then replace the core soft library with the built macro for a hierarchical flow.
You can use the write_dependencygraph to view the siliconcompiler library dependencies.

I think i got it to work. Thank you for the pointers!

@Ciprian167 yes, you should be able to run them all in parallel locally (just watch your memory since that can sometimes become an issue).

There are two possible reasons it's not running in parallel:

1. (most likely) if you have not specified the number of threads a node uses, we assume it's all your cores. Therefore if you know it's only using 1 thread, you can set tool,<tool>,task,<task>,threads = 1 (https://docs.siliconcompiler.com/en/latest/reference_manual/schema.html#param-tool-task-threads-ref), and they will allow the scheduler which limits the parallel processes to execute more than 1 at a time. You can also set (https://docs.siliconcompiler.com/en/latest/reference_manual/schema.html#param-option-scheduler-maxnodes-ref) to limit the total number of nodes running in parallel.
2. (less likely) if you manually specified some dependency between the nodes, then they cannot be parallelized.

It works! Thank you. I was missing the tool,<tool>,task,<task>,threads config. I wasn't able to infer that, and it's not explicitly stated anywhere. But now it makes sense.

So when setting chip.set('option', 'scheduler', 'name', 'slurm') will it replace this sc scheduler with slurm, or will there be two scheduling levels, one for the flowgraph and the slurm one?

two levels, in both cases the sc flowgraph is executed normally, but with slurm turned on the individual nodes will be dispatched via slurm and slurm handles executing the nodes.

Got it. Thank you very much!