Linux oven
This page documents the proposal for a new tree for the future and tools enhancements to help with cross tree collateral evolutions for Linux development. Without advanced tooling developed it may perhaps be premature for this tree to exist.
Contents
The case for linux-oven
The term collateral evolution was formally coined first on the paper Understanding Collateral Evolution in Linux Device Drivers in 2005 as "changes in the interfaces exported by the kernel and driver support libraries". The goal behind the Coccinelle engine, also described first in the paper was to help make these sorts of changes easier on the Linux kernel. Coccinelle has picked up but its just one way in which collateral evolutions are often made these days on Linux, other times perl, and sed scripts are used. Thanks to scripting and tools such Coccinelle these days doing collateral evolutions is a much more trivial task, however for Linux one still needs to deal with subsystem maintainers. If a 'collateral evolution' is to be applied to one subsystem the series of patches that deal with that 'collateral evolution' can typically be coordinated with the subsystem maintainer and this typically is just a matter of a few e-mail exchanges. Collateral evolutions that require changes over a series of subsystems requires quite a bit more work though. There are two strategies that subsystem maintainers have typically chosen for this:
- Submit changes to Linus at the beginning of the merge window
- Submit changes to Linus at the end of the merge window
Who sends the changes out can vary, preferably this is done by a subsystem maintainer, however issues can still arise. If we had a separate tree and a dedicated maintainer to help with this it should help make the process smoother. To make it even easier though we should document what types of scripts are welcomed, and types of tests are needed prior to submitting to this tree.
The 'linux-oven' subsystem tree is currently just a proposal for the future, once the other details of tooling are ironed out. This topic was first raised during the South Korea Linux kernel summit in 2014 as a proposed agenda item topic. Linus' response to the topic was that "we currently don't have any good solution to this". This proposal tries to pave a way for a formal solution to the problem.
linux-oven based on linux-next
The linux-next tree already helps the merge window process by merging all pending changes which we envision will be submitted to Linus during the next merge window typically daily. Its a scripted Linus, if you will. To help address cross-tree collateral evolutions a tree could be based on linux-next as then it would include the state of Linux as most subsystem developer envision it for the next merge window. As a last step, it could then apply the series of pending scripted changes. These scripts would never be upstream.
The linux-next tree then could avoid the linux-oven, linux-oven would only have cross tree collateral evolution changes. To rebase changes the linux-oven tree would then have issued:
- git fetch linux-next git checkout -b 20170628-oven 20170628-next
- /run-some-tools.sh
Supported tooling
- sed
- perl
- coccinelle
Tool enhancements
Below are ideas to help cope with the automation of maintenance of cross tree collateral evolutions.
Coccinelle enhancements
This lists a few possible enhancements to either Coccinelle or related tools to use Coccinelle on Linux.
Coccinelle rebase via git
Linux kernel changes that use Coccinelle often describe that the Coccinelle engine was used to make the changes. Lately, mostly upon request by other curious Coccinelle developers, these type of changes also have embedded on them in the commit log the respective Coccinelle SmPL code used. There is one strong exception: changes that make use of SmPL patches which are already merged upstream on the Linux kernel's scripts/coccinelle/ directory. Nevertheless one can typically find the respective set of SmPL grammar rules used for changes.
If changes are to be submitted at the end of a merge window it would mean the linux-oven tree would need to be rebased regularly, perhaps bi-weekly, it'd be based on linux-next. Since linux-oven would be based on linux-next naturally it would prefer if the merge order followed would match the order of scripted merges on linux-next, but this is not necessarily required. Since linux-oven would be rebased regularly to help cope with changes in other subsystems and avoid issues with having to manually fix conflicting hunks git could be extended to enable a series of options to deal with patch failures. One of the mechanisms could be to read the commit log, determine if Coccinelle was used and if it was as way to resolve the hunk conflict drop the patch and now re-apply it again with Coccinelle.
This would only work with patches that are *completely* expressible with SmPL, however some collateral evolutions that make use of SmPL may require additional changes which are expressed best by regular diff, we refer to this as patch addendum. It is being considered whether or not and how SmPL could incorporate the expression of patch addendums, and if it and how to encode this on commits.
To help tooling (git) figure out if a patch was based on Coccinelle a tag can be used. Some tags already are in place, but they are not consistent. To help with this prospect for future development one can use this tag:
- Generated-by: Coccinelle SmPL
SmPL inference support
Not all developers are comfortable with writing rules in SmPL grammar for Coccinelle. To account for this if Coccinelle had a way to infer SmPL grammar and perhaps a patch addendum (defined above), then rebasing via git could probably be more useful. This is being considered.
Interested or volunteered developers
- Luis Rodriguez
- Julia Lawall
Needed developers
If anyone out there is interested in developing a generic git hook to enable addressing patch conflicts with custom possible resolutions please list your name here and contact the interested developers / volunteers above.