Not sure anything can be done about this, but just leaving a note in case @jklymak (or someone else) in interested in working on the performance: with figure.constrained_layout.use set (mpl2.2.0rc1), on my machine,

• subplots(4, 4) takes 4s,
• subplots(5, 5) takes 28s,
• subplots(6, 6) takes 48s.