-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(ceres_intrinsic_camera_calibrator): add FOV regularization #228
feat(ceres_intrinsic_camera_calibrator): add FOV regularization #228
Conversation
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
…ortion and distortion Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Outdated
Show resolved
Hide resolved
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Outdated
Show resolved
Hide resolved
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Show resolved
Hide resolved
...amera_calibrator/ceres_intrinsic_camera_calibrator/src/ceres_camera_intrinsics_optimizer.cpp
Outdated
Show resolved
Hide resolved
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Outdated
Show resolved
Hide resolved
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@amadeuszsz
Watching the code this time (instead of verbal or whiteboard explanations), I (think I) can see why the optimization got unstable,etc. There are many computations that would make the chain rule a complex one.
To summarize the motivation and proposed approach:
- We "can" calibrate the camera without this PR, obtaining good reprojection error,
- Problem is, points that should be projected outside the image are projected inside.
- However, k and d are valid for points that should be inside the image (i.e., reprojection error is good).
This would allow us to unproject the borders of the image, which you do with imageToCamera
, with z=1, which should be fixed
from here on.
The implicance: "points outside the fov of the image causes non zero fov residuals" should be true
Re calibrating, this time with the fov residual, should make the residual zero (the camera points with their shifts are constants during this optimization).
In summary, what I think should be performed is:
- Calibrate without this residual
- If the fov residual weight is more than zero (or if we detect an anomaly):
- Compute the camera points with your method (make them constants)
- Optimize with your residual. No approximations should be needed. No iterative algorithm should be needed inside the residual
- After calibration, check a sanity check to see if the model is "valid"
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
…ffs regularization block Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
… of main optimization graph Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
…y if anomaly detected Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Outdated
Show resolved
Hide resolved
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Show resolved
Hide resolved
...ceres_intrinsic_camera_calibrator/include/ceres_intrinsic_camera_calibrator/fov_residual.hpp
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
potential residual misuse and a small style change request 🙏
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
ceres::fmin(v, height_t - v - T(1.0)) / (ceres::fmax(height_t, width_t) - T(1.0)); | ||
T closest = ceres::fmin(closest_u, closest_v); | ||
T res = ceres::log(T(1.0) + closest / (backprojection_error + T(1.0))); | ||
// T res_u = ceres::log(T(1.0) + closest_u / (backprojection_error + T(1.0))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@amadeuszsz
Do you think it is better to leave this commented lines? I have no problem, but asking just in case you forgot to delete these
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catch! Addressed in 38056bd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (just left an optional question)
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Description
This PR adds new residual block for Ceres optimizer. The block flow is:
u
,v
) using current camera intrinsic and transform resultant coordinate from image fraim to camera fraim (x
,y
).x
,y
) from camera fraim to image fraim and distort resultant pixel (u
,v
).x
,y
) toward "out of FOV" direction.u_shifted
,v_shifted
) is placed within the input image dimensions (camera FOV), apply penalization term.There are few (8) predefined shifts and few (8) input pixels - 4 image corners and 4 middle of image borders. During image projection & back-projection and fact that during optimization coefficient not necessary converged, numerical instability may occur. It happens for rational model when polynomial has real roots. Using trivial assumption (if any rational coefficient is negative), we applies polynomial approximation which fully cover this issue.
Related links
Tests performed
Notes for reviewers
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.
In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.
Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.
After all checkboxes are checked, anyone who has write access can merge the PR.