-
-
Notifications
You must be signed in to change notification settings - Fork 32.4k
bpo-45876: Correctly rounded stdev() and pstdev() for the Decimal case #29828
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
bbd2da9
74bdf1b
6c53f1a
a487c4f
eb56423
cc7ba06
d024dd0
b10f912
fb6744d
7f21a1c
7da42d4
e31757b
f058a6f
1fc29bd
e5c0184
3c86ec1
96675e4
de558c6
418a07f
ea23a8b
ba248b7
037b5fe
f5c091c
70cdade
82dbec6
1a6c58d
b2385b0
594ea27
3911581
a09e3c4
152ed3f
80371c1
1c86e7c
0684fac
8b5e377
d11d567
309cb0a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -334,24 +334,27 @@ def _decimal_sqrt_of_frac(n: int, m: int) -> Decimal: | |
"""Square root of n/m as a float, correctly rounded.""" | ||
# Premise: For decimal, computing (n/m).sqrt() can be off by 1 ulp. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Unfortunately, while the corresponding result is true for binary floating-point, it's not true for decimal floating-point: for decimal, the error can be more than 1 ulp. Example using 3-digit precision: >>> from decimal import Decimal, getcontext
>>> n, m = 209, 20
>>> getcontext().prec = 3
>>> (Decimal(n) / Decimal(m)).sqrt()
Decimal('3.22')
>>> getcontext().prec = 10
>>> (Decimal(n) / Decimal(m)).sqrt()
Decimal('3.232645975') The correctly-rounded value here is around 1.265 ulps away, at 3.232645975... However, I believe that one can show that the worst case error is at most 1.3ulp (actually 0.5 + sqrt(10)/4 = 1.290569... ulps), in which case it remains true that the correctly-rounded result will always be one of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Self-correction: I guess "off by <= 1ulp" is accurate, so long as it's clear that we're talking about the maximum difference between the computed value and the correctly-rounded value. I was thinking in terms of the max difference between the computed value and the true mathematical value. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updating the comment. |
||
# Method: Check the result, moving up or down a step if needed. | ||
if not n: | ||
return 0.0 | ||
f_square = Fraction(n, m) | ||
if n <= 0: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we be checking There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The case where m is zero is handled before the inequality test. The There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I was really more worried about the case m negative, not m zero. It seems more natural to have the normalization step It doesn't matter, because if just one of |
||
if not n: | ||
return 0.0 | ||
n, m = -n, -m | ||
|
||
d_start = (Decimal(n) / Decimal(m)).sqrt() | ||
f_start = Fraction(*d_start.as_integer_ratio()) | ||
root = (Decimal(n) / Decimal(m)).sqrt() | ||
nr, dr = root.as_integer_ratio() | ||
|
||
d_plus = d_start.next_plus() | ||
f_plus = Fraction(*d_plus.as_integer_ratio()) | ||
if f_square > ((f_start + f_plus) / 2) ** 2: | ||
return d_plus | ||
plus = root.next_plus() | ||
np, dp = plus.as_integer_ratio() | ||
# test: n / m > ((root + plus) / 2) ** 2 | ||
if 4*dr**2*dp**2*n > m*(dr*np + dp*nr)**2: | ||
return plus | ||
|
||
d_minus = d_start.next_minus() | ||
f_minus = Fraction(*d_minus.as_integer_ratio()) | ||
if f_square < ((f_start + f_minus) / 2) ** 2: | ||
return d_minus | ||
minus = root.next_minus() | ||
nm, dm = minus.as_integer_ratio() | ||
# test: n / m < ((root + minus) / 2) ** 2 | ||
if 4*dr**2*dm**2*n < m*(dr*nm + dm*nr)**2: | ||
return minus | ||
|
||
return d_start | ||
return root | ||
|
||
|
||
# === Measures of central tendency (averages) === | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
copypasta: "as a float"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.