Feature or enhancement

Enable instruction definitions for generate_cases.py to be spread across multiple files.

Pitch

This would allow projects like Cinder to alter/add instructions while still being able to use the base bytecodes.c without modification. Thus we can minimize the amount of conflict when merging upstream, and avoid copying/forked code if Cinder's features are moved into a standalone module.

In the attached PR I've put together an implementation that allows multiple inputs to be passed to generate_cases.py by using -i multiple times. E.g.:

$ generate_cases.py -i bytecodes.c -i cinder-bytecodes.c -o generated.c.h

The input files are mostly processed as if they were concatenated together, although for each file parsing is only carried out between the BEGIN/END BYTECODES markers.

I've also added a new override keyword which can prefix instruction definitions to indicate the intent to completely replace an existing definition. E.g.:

inst(NOP, (--)) {
}

// Only this version is used during generation.
override inst(NOP, (--)) {
  magic();
}

// Error - NOP already exists but "override" not specified.
inst(NOP, (--)) {
  magic();
}

// Error - no existing ZOP to override.
override inst(ZOP, (--)) {
}

Overrides allow us to introduce changes to a small number of instructions which currently need to behave slightly differently in Cinder. The override keyword allows the parser to both enforce only explicitly allowed overrides occur, and there are no "false" overrides. This will quickly reveal if a new instruction is introduced in the base definition which unexpectedly conflicts with something Cinder added, and similarly if something Cinder overrode is removed.

Previous discussion

The idea of using multiple files to generate the interpreter was briefly discussed around the faster-cpython project.