Equation for scores_ in BayesianRidge seems incorrect. #10748
Description
Description
In class BayesianRidge, the formula for computing score (s) (this is the objective or log marginal likelihood) is incorrect.
The sign before logdet_sigma is a '-' and it should be '+' as per reference slides (page 32, last equation) -> R. Salakhutdinov, Lecture notes on Statistical Machine Learning,
http://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=15
Their beta is our self.alpha_
Their alpha is our self.lambda_
Actual Code (lines 218-227)
,# Compute the objective function
if self.compute_score:
s = lambda_1 * log(lambda_) - lambda_2 * lambda_
s += alpha_1 * log(alpha_) - alpha_2 * alpha_
s += 0.5 * (n_features * log(lambda_) +
n_samples * log(alpha_) -
alpha_ * rmse_ -
(lambda_ * np.sum(coef_ ** 2)) - # <- wrong sign here '-' before logdet_sigma
logdet_sigma_ -
n_samples * log(2 * np.pi))>
Expected Code
,# Compute the objective function
if self.compute_score:
s = lambda_1 * log(lambda_) - lambda_2 * lambda_
s += alpha_1 * log(alpha_) - alpha_2 * alpha_
s += 0.5 * (n_features * log(lambda_) +
n_samples * log(alpha_) -
alpha_ * rmse_ -
(lambda_ * np.sum(coef_ ** 2)) + # <- should be '+' before logdet_sigma
logdet_sigma_ -
n_samples * log(2 * np.pi))>
This is the snapshot of the marginal log likelihood equation from the reference, the sign before (their S_{N} is your sigma_ ) 0.5*logdet_sigma is + not -.
