Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ENH log1pexp for binomial loss in loss module #21814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 30, 2021
Merged

ENH log1pexp for binomial loss in loss module #21814

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d703fa2
ENH log1pexp
lorentzenchr Nov 29, 2021
3c3faa2
MNT set loss class default to n_classes=None
lorentzenchr Nov 29, 2021
e1b9179
MNT comment for reason of further cuttof at -2
lorentzenchr Nov 29, 2021
9400640
DOC wording: accurate instead of precise
lorentzenchr Nov 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions sklearn/_loss/_loss.pyx.tp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,13 +203,19 @@ np.import_array()
# -------------------------------------
# Helper functions
# -------------------------------------
# Numerically stable version of log(1 + exp(x)) for double precision
# See https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
# Numerically stable version of log(1 + exp(x)) for double precision, see Eq. (10) of
# https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf
# Note: The only important cutoff is at x = 18. All others are to save computation
# time. Compared to the reference, we add the additional case distiction x <= -2 in
# order to use log instead of log1p for improved performance. As with the other
# cutoffs, this is accurate within machine precision of double.
cdef inline double log1pexp(double x) nogil:
if x <= -37:
return exp(x)
elif x <= 18:
elif x <= -2:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How was -2 chosen here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add a shorter note.
The longer story is at the end of section 2 in https://cran.r-project.org/web/packages/Rmpfr/vignettes/log1mexp-note.pdf, arguing for log(2) as cutoff for the function log(1-exp(-x)).
I tested this -2 on our function and it gives a difference of 1e-16.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

import numpy as np

def diff(x):
    """Return abs diff and rel diff"""
    use_log = np.log(1 + np.exp(x))
    use_log1p = np.log1p(np.exp(x))
    return use_log - use_log1p, use_log / use_log1p - 1

for x in [0, -1, -2, -3]:
    print(f"x={x}: {diff(x)}")

results in

x=0: (0.0, 0.0)
x=-1: (0.0, 0.0)
x=-2: (1.1102230246251565e-16, 8.881784197001252e-16)
x=-3: (-1.0408340855860843e-16, -2.1094237467877974e-15)

return log1p(exp(x))
elif x <= 18:
return log(1. + exp(x))
elif x <= 33.3:
return x + exp(-x)
else:
Expand Down
2 changes: 1 addition & 1 deletion sklearn/_loss/loss.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ class BaseLoss:
differentiable = True
is_multiclass = False

def __init__(self, closs, link, n_classes=1):
def __init__(self, closs, link, n_classes=None):
self.closs = closs
self.link = link
self.approx_hessian = False
Expand Down