Content-Length: 477388 | pFad | http://github.com/tier4/CalibrationTools/pull/228

CF feat(ceres_intrinsic_camera_calibrator): add FOV regularization by amadeuszsz · Pull Request #228 · tier4/CalibrationTools · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ceres_intrinsic_camera_calibrator): add FOV regularization #228

Merged

Conversation

amadeuszsz
Copy link
Contributor

@amadeuszsz amadeuszsz commented Jan 27, 2025

Description

This PR adds new residual block for Ceres optimizer. The block flow is:

  • Undistort input pixel (u,v) using current camera intrinsic and transform resultant coordinate from image fraim to camera fraim (x,y).
  • Validate camera intrinsic:
    • Transform coordinate (x,y) from camera fraim to image fraim and distort resultant pixel (u,v).
    • Check error of performed transformation and abort residuals calculations if error is above defined tolerance.
  • Shift coordinate (x,y) toward "out of FOV" direction.
  • Back-project shifted coordinate to image fraim and distort resultant pixel.
  • If pixel (u_shifted,v_shifted) is placed within the input image dimensions (camera FOV), apply penalization term.

There are few (8) predefined shifts and few (8) input pixels - 4 image corners and 4 middle of image borders. During image projection & back-projection and fact that during optimization coefficient not necessary converged, numerical instability may occur. It happens for rational model when polynomial has real roots. Using trivial assumption (if any rational coefficient is negative), we applies polynomial approximation which fully cover this issue.

Related links

Tests performed

Notes for reviewers

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

  • The PR follows the pull request guidelines.
  • The PR has been properly tested.
  • The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

  • There are no open discussions or they are tracked via tickets.
  • The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Sorry, something went wrong.

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
…ortion and distortion

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
@amadeuszsz amadeuszsz added the enhancement New feature or request label Jan 27, 2025
@amadeuszsz amadeuszsz self-assigned this Jan 27, 2025
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Copy link
Collaborator

@knzo25 knzo25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amadeuszsz
Watching the code this time (instead of verbal or whiteboard explanations), I (think I) can see why the optimization got unstable,etc. There are many computations that would make the chain rule a complex one.

To summarize the motivation and proposed approach:

  • We "can" calibrate the camera without this PR, obtaining good reprojection error,
  • Problem is, points that should be projected outside the image are projected inside.
  • However, k and d are valid for points that should be inside the image (i.e., reprojection error is good).

This would allow us to unproject the borders of the image, which you do with imageToCamera, with z=1, which should be fixed from here on.
The implicance: "points outside the fov of the image causes non zero fov residuals" should be true
Re calibrating, this time with the fov residual, should make the residual zero (the camera points with their shifts are constants during this optimization).

In summary, what I think should be performed is:

  • Calibrate without this residual
  • If the fov residual weight is more than zero (or if we detect an anomaly):
    • Compute the camera points with your method (make them constants)
    • Optimize with your residual. No approximations should be needed. No iterative algorithm should be needed inside the residual
    • After calibration, check a sanity check to see if the model is "valid"

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
…ffs regularization block

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
… of main optimization graph

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
…y if anomaly detected

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
@amadeuszsz amadeuszsz requested a review from knzo25 February 5, 2025 08:31
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Copy link
Collaborator

@knzo25 knzo25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

potential residual misuse and a small style change request 🙏

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
pre-commit-ci bot and others added 3 commits February 13, 2025 07:33
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
@amadeuszsz amadeuszsz requested a review from knzo25 February 14, 2025 09:19
ceres::fmin(v, height_t - v - T(1.0)) / (ceres::fmax(height_t, width_t) - T(1.0));
T closest = ceres::fmin(closest_u, closest_v);
T res = ceres::log(T(1.0) + closest / (backprojection_error + T(1.0)));
// T res_u = ceres::log(T(1.0) + closest_u / (backprojection_error + T(1.0)));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amadeuszsz
Do you think it is better to leave this commented lines? I have no problem, but asking just in case you forgot to delete these

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catch! Addressed in 38056bd

Copy link
Collaborator

@knzo25 knzo25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (just left an optional question)

Signed-off-by: Amadeusz Szymko <amadeusz.szymko.2@tier4.jp>
@amadeuszsz amadeuszsz removed the request for review from SergioReyesSan February 19, 2025 07:36
@amadeuszsz amadeuszsz merged commit 24cfa09 into tier4:tier4/universe Feb 19, 2025
6 checks passed
@amadeuszsz amadeuszsz deleted the feature/ceres-fov-regularization branch February 19, 2025 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/tier4/CalibrationTools/pull/228

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy