Hi, I am interested in working on this issue! I have experience in image processing and calibration with OpenCV.
I will investigate the undistortPoints() function in detail and try to reproduce the issue with test cases.
Can you confirm if anyone else is already working on this?
Looking forward to contributing!

When I tried to transform some image coordinates within the area of image of a camera to world coordinates using undistortPoints and then back to image coordinates using projectPoints, I could get coordinates very close to the ones before transformation (undistortPoints) in most cases but could not get a close coordinates for the ones far from the image center. Here, I used undistortPoints and projectPoints both in https://github.com/opencv/opencv/blob/4.x/modules/calib3d/src/fisheye.cpp. I tried to find the cause and reached the conclusion that it is due to the implementation of undistortPoints, because when I implemented locally an alternative function for undistortPoints (in Python) which meets the following policies, I could get coordinates very close to the ones before transformation even in cases where the image points are far from the image center.

Policies:

1. maximum value of theta_d should not be pi/2
   Because theta_d is not an angle, the maximum value should not be limited to pi/2.
   Instead the maximum value should be limited to the value derived from the following equation using pi/2 for theta
   , which I call "max_theta_d" in the following discussion.
   
   Looking at (https://github.com/opencv/opencv/blob/4.x/modules/calib3d/src/fisheye.cpp#L434) it is obvious that
   theta_d derived >= 0, so overall theta_d should meet 0 <= theta_d <= max_theta_d and should be limited likewise.
   (We don't have to care about negative values when using the value derived by L434.)
   We can understand that theta_d is not an angle by going through the following documents:
   https://docs.opencv.org/4.11.0/db/d58/group__calib3d__fisheye.html
   https://docs.opencv.org/4.11.0/d0/de3/citelist.html#CITEREF_kannala2006

2. theta should meet 0 <= theta <= pi/2
   In undistortPoints, theta is initialized to theta_d
   (https://github.com/opencv/opencv/blob/4.x/modules/calib3d/src/fisheye.cpp#L442)
   , which could be greater than pi/2 after following the policy 1 above. Because theta is an angle it should meet
   0 <= theta <= pi/2, and should be limited likewise.

Because I don't have the skill to implement the above in C++, I would be very grateful if someone skilled enough implemented considering the above.

Actually, in my local implementation, I tried initializing theta with the following value which lead to a quicker convergence.
theta = theta_d / ratio_max_thetas
which is equivalent to
theta_d = ratio_max_thetas * theta
where
ratio_max_thetas = max_theta_d / (pi/2)
I believe this initialization is better for cameras in which the relationship between theta and theta_d can be roughly approximated by
theta_d = ratio_max_thetas * theta (as in the following figure)

I believe it will be beneficial if the initialization described above can be chosen as an option. Personally, I think this initialization could also serve as a default initialization. I'll be grateful if this is considered when implementing.

Hi there I briefly wanted to chime in. I fixed the issue with #27199.
As mentioned rightfully by @Tomoyuki-Ohtsuki, theta_d not a proper angle but is equal to the radius of the distorted pixel.

The routine in undistortPoint inverts the fisheye distortion by solving the following equation:

Therefore:

Only the solution thetas need to be between 0 and pi/2.
For further reference read: Kannala-Brandt camera model

Also here is a sanity check, which undistorts and redistorts a regular image grid:

import cv2
import numpy as np
import matplotlib.pyplot as plt

h, w = 384, 576
xv, yv = np.meshgrid(np.arange(w), np.arange(h))
distorted_points = np.vstack([xv.ravel(), yv.ravel()]).T.astype("float32")  # (N,2)
intrinsics = np.array([2.0268e+02,  2.0289e+02,  2.8999e+02,  1.9243e+02,  5.6743e-02, -4.8614e-03, -1.1769e-04, -2.9011e-04])
K = np.zeros((3,3))
K[0,0] = intrinsics[0]
K[1,1] = intrinsics[1]
K[0,2] = intrinsics[2]
K[1,2] = intrinsics[3]
K[2,2] = 1

undistorted_points = cv2.fisheye.undistortPoints(
    distorted=distorted_points[:, np.newaxis, :],
    K=K,
    D=intrinsics[4:],
    R=None,
    P=np.eye(3)
)

plt.scatter(undistorted_points[:,0,0], undistorted_points[:,0,1])
plt.axis("equal")
plt.show()

redistorted_points = cv2.fisheye.distortPoints(
    undistorted=undistorted_points,
    K=np.eye(3),
    D=intrinsics[4:],
)
plt.scatter(redistorted_points[:,0,0], redistorted_points[:,0,1], color='red')
plt.axis("equal")
plt.show()

Before Fix (=> Clamped edges) :

After Fix:

Note: Yes, the undistortion really stretches out the pixels. The furthest point is undistorted to (-121.6, -80.6). We can calculate the angle in degrees as follows:

print(np.degrees(np.arctan(np.linalg.norm(undistorted_points[0]))))

This prints 89.60795, which corresponds to a FOV to 179.215 degrees.

Values which are not in the valid range are set to (-1000000.0, -1000000.0) as before.

EDIT: Note for @gitcheol. I don't think the incident angle is sqrt(atan2(theta_d**2+1)/theta_d)). I think we can only solve for the incident angle by using the root finder. Usually atan(r) = theta only holds for the undistorted (rectified) points. But feel free to correct me.

Hope this helps.
Kind regards

@Tomoyuki-Ohtsuki I also tried implementing your strategy but found only little difference in terms of average iterations. Do you have some more examples I could try?

@mokrueger

Thank you so much for fixing the problem!
Inspired by your comment, I re-analyzed what's happening when it takes a lot of iteration to converge (or finish without convergence by reaching maximum iteration, which I set to 1000 this time) when using theta_d as the initial value of theta. And I
found out that salient cases are caused by unexpected situation, which I explain below.
When using the following values for distortion

k1 = 0.07394581288099289
k2 = 0.05758174881339073
k3 = -0.02578509785234928
k4 = 0.004517474211752415

somehow, I get negative values for inclination of theta_d, which is calculated by

1 + 3 * k0_theta2 + 5 * k1_theta4 + 7 * k2_theta6 + 9 * k3_theta8

(found in https://github.com/opencv/opencv/blob/4.x/modules/calib3d/src/fisheye.cpp).

The followings are plots of theta_d and inclination of theta_d (theta_d') and, weirdly, looking at the plots of theta_d, it doesn't look like it has a negative inclination.

Due to the fact that inclination is calculated as a negative value, in many cases, when I executed the calculation, it went through an infinite loop, resulting in reaching the maximum iteration. I post an example for the case where theta_d = 1.5653739337616015 and the initial value of theta is theta_d.

theta before update: 1.5653739337616015
theta_fix: -0.7832073433322058
theta after update: 2.3485812770938073
theta after clipping: 1.5707963267948966

theta before update: 1.5707963267948966
theta_fix: -0.6329056387529547
theta after update: 2.203701965547851
theta after clipping: 1.5707963267948966

theta before update: 1.5707963267948966
theta_fix: -0.6329056387529547
theta after update: 2.203701965547851
theta after clipping: 1.5707963267948966

repetition of the above 4 lines continues

theta after update = theta before update - theta_fix
theta is clipped by pi/2

This infinite loop does not happen when I use the initial value that I proposed.

I wonder why I get negative value of inclination which lead to non-convergence of Newton method. It's possible that this is true only in the case of python but I haven't verified.
By looking at the result of the above camera (distortion), I believed that my proposal would lead to much quicker convergence .
Sorry for the improper explanation I posted.

I, actually, analyzed a case of another camera (distortion) where

k1 = 0.09150316566228867
k2 = 0.035320695489645004
k3 = -0.01622619479894638
k4 = 0.0035546349827200174 .

In this case, for example, the numbers of iteration were as follows for theta_d = 2.0245863572631397 .

546 for initial theta = theta_d
538 for initial theta is what I proposed

The improvement is much smaller compared to the first camera, but I still believe it is worth implementing because I assume it achieves quicker convergence for a major part of existing fisheye cameras.
I would be happy if you could consider implementing the proposed initial value.
Thanks again.

@Tomoyuki-Ohtsuki How did you calculate that the derivative / theta_fix is negative? I plotted both functions and they should never produce something negative unless theta_d is negative(? I think). Can you show me how you logged it the values. Also I currently don't understand why it there is clipping inside the loop? Could you clarify. Any kind of code snippet would also be great such that I can properly debug it 👍

EDIT: Here's a plot of f(x) and the derivative with your first example:

So I currently cannot see how for 1.57 theta_fix could be negative

EDIT2: Heres a sympy snippet with the calculation:

from sympy import is_monotonic, symbols, Interval, pi, diff
theta = symbols('theta')
k1 = 0.07394581288099289
k2 = 0.05758174881339073
k3 = -0.02578509785234928
k4 = 0.004517474211752415
theta_d = 1.5653739337616015

theta_curr = theta_d
f_theta = theta + k1 * theta ** 3 + k2 * theta ** 5 + k3 * theta ** 7 + k4 * theta ** 9 - theta_d
f_diff = diff(f_theta)
theta_fix = f_theta.subs('theta', theta_d) / f_diff.subs('theta', theta_d)
print(f"before update : {theta_curr}")
theta_curr = theta_curr - theta_fix
print(f"theta_fix : {theta_fix}")
print(f"after update : {theta_curr}")

Prints:

before update : 1.5653739337616015
theta_fix : 0.233347280712034
after update : 1.33202665304957

(Note that the solution for this theta_d lies at 1.302)

@mokrueger
I'm terribly sorry. I found a bug in my code which was the cause of non-convergence and negative derivative. After fixing the code I could achieve convergence for any value I tried for both cameras. I also got similar plots for theta_d and theta_d' as yours.
The number of iteration that I got so far from some experiments are around 4 to 5 and does not look to worsen even if I used theta_d as initial value for theta. So I don't insist on asking to implement my proposal for the initial value of theta, though I simply feel it's more beautiful for a major part of existing cameras.
I put a snippet of my code of "undistortPoints" below:

MAX_COUNT = 1000
CRITERIA_EPS = 1e-8

...

            for j in range(MAX_COUNT):
                theta2 = theta**2
                theta4 = theta**4
                theta6 = theta**6
                theta8 = theta**8
                k0_theta2 = D[0] * theta2
                k1_theta4 = D[1] * theta4
                k2_theta6 = D[2] * theta6
                k3_theta8 = D[3] * theta8
                theta_fix = (
                    theta * (1 + k0_theta2 + k1_theta4 + k2_theta6 + k3_theta8)
                    - theta_d
                ) / (1 + 3 * k0_theta2 + 5 * k1_theta4 + 7 * k2_theta6 + 9 * k3_theta8)
                print(f"theta before update: {theta}")
                print(f"theta fix: {theta_fix}")
                theta -= theta_fix
                print(f"theta after update: {theta}")
                if theta > math.pi / 2.0:
                    theta = math.pi / 2.0
                    print(f"theta after clipping: {theta}")
                if math.fabs(theta_fix) < CRITERIA_EPS:
                    break

D contains distortion coefficients.
I clip the value of theta to pi/2 at maximum right before the checking of convergence, which can be found 3 and 4 lines from the bottom.
I do this because, in this function (undistortPoints), I assume that theta is defined in the range of 0 <= theta <= pi/2, and so theta_d is not guaranteed to be monotonically increasing with respect to theta for theta > pi/2. When theta_d is not monotonically increasing, the above calculation might not converge, which I accidentally illustrated with my buggy code.
Thanks!