It's more an idea than an issue, but maybe using DragonFFI (github.com/aguinet/dragonffi) as a compiler and "python glue" (that is convert data back and forth from C and python) backend can solve some of the issues you have (like generating many .so's).

Let me know if you are interested!