In order for @spall to integrate their work with a different scheduler, it's important to have more than one scheduler. I initially suggested a free monad, but I don't think that's the right abstraction anymore. Thinking about it:

• The user issues commands. There are basically two things the user can do that impact the scheduler in any way - they can call parallel to run many commands in parallel, and call cmd to cause an action to run an action.
• The scheduler can run at most N commands at a time. At each time step, the scheduler has to decide which cmd to run.
• After a cmd completes, the scheduler is responsible for deciding if it got a hazard, or not, and what to do next.

So my questions are:

• Should the scheduler have access to the parallel information? Or should these just look like commands and the scheduler is unaware of their topology of branching? @spall - do you think you can make use of that information?
• Do we want to enable cancelling a command halfway through?
• Do we ever want to store more than just "what happened last time" in the speculation?

I am thinking a "scheduler" should be abstract in what a command is, and roughly have the interface:

-- given c which is the abstract type of command
-- and s which is the abstract type of information stored in a scheduler
newScheduler :: [(c, RW)] -> s -- create based on what happened last time
scheduleDequeue :: s -> Maybe (c, s) -- what would the scheduler like to execute next
scheduleEnqueue :: c -> s -> s -- something has been demanded by the user
scheduleRetire :: c -> RW -> s -> Either FatalError s -- something finished executing

That doesn't let you cancel running jobs, and doesn't let you observe where the user requests parallelism. My guess is s would be a type class, so we would have multiple schedulers to choose from, and c would be polymorphic so that the scheduler can only use c as a key.