ENH Newton-CD part 5: add NewtonCDGramSolver for L1-penalized GLMs - #34377

Reference Issues/PRs

Part 5 of #33160.

What does this implement/fix? Explain your changes.

This PR implements the new GLM-solver class NewtonCDGramSolver. It uses our coordinate-descent function enet_coordinate_descent_gram under the hood as inner solver.

In this PR, the solver="newton-cd-gram" is publicly exposed in LogisticRegression and LogisticRegressionCV, but not yet (only privately) in PoissonRegressor, GammaRegressor, TweedieRegressor. The reason is that for the latter, a new parameter l1_ratio needs to be introduced. Not doing so keeps this PR a bit smaller.

The tests are quite rigorous in my opinion and they pass.

The mechanics of a Newton Solver:

• Outer loop
  • Compute gradients g and hessians H
  • Inner loop:
    Compute a Newton step H @ step = -g
    This solver uses coordinate descent.
  • Line search (guarantee global convergence)
  • Check for convergenge
• End outer loop

AI usage disclosure

None

Any other comments?

This solver is good for n_features << n_samples.