Nice work on this, it's clear that there is quite some more information to get from the imagery. But I have two questions:

• in the recipes you use solar_pathlength_corrected, but where is this defined? I can only find effective_solar_pathlength_corrected defined here https://github.com/pytroll/satpy/blob/main/satpy/etc/composites/visir.yaml#L14.
• Could you also provide a couple of example of how your proposed changes look in the day-night blends, i.e. cloud_type_with_night_ir105 and cloud_phase_with_night_ir105? We may need to tweak the blending there as well.

And as a comment, I still find that at least for the thick part of the frontal cloud close to the terminator the level of saturation is a bit too high. But I'm also aware that it's hard to find settings that works well for all conditions.

• in the recipes you use solar_pathlength_corrected, but where is this defined?

Oops. Only in my local recipes. Fixed with 6d5fe19. It shows why we need the automated imagery tests, and probably we should indeed merge #3013 first (where I added more on the behave tests) so I can leverage this!

Will check the versions with night_ir105.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.22%. Comparing base (78e88c4) to head (abd4f69).
Report is 41 commits behind head on main.

@@            Coverage Diff             @@
##             main    #3032      +/-   ##
==========================================
- Coverage   96.23%   96.22%   -0.02%     
==========================================
  Files         398      398              
  Lines       57325    57382      +57     
==========================================
+ Hits        55168    55217      +49     
- Misses       2157     2165       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Pull Request Test Coverage Report for Build 15068599121

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 18 unchanged lines in 6 files lost coverage.
• Overall coverage decreased (-0.01%) to 96.329%

Totals
Change from base Build 14862404225:	-0.01%
Covered Lines:	55477
Relevant Lines:	57591

💛 - Coveralls

Cloud type night IR 10.5, before:

Cloud type night IR, after:

Cloud phase night IR, before:

Cloud phase night IR, after:

For curiosity, our internal solar_pathlength_corrected, that I accidentally used in the first commit, is defined as:

  solar_pathlength_corrected:
    modifier: !!python/name:satpy.modifiers.EffectiveSolarPathLengthCorrector
    optional_prerequisites:
      - solar_zenith_angle
    max_sza: !!null

whereas the satpy one is defined as:

The version I showed in the initial issue description is with max_sza: !!null, which we use internally for all our composites.

Cloud type with the effective solar pathlength corrector, default max_sza:

Cloud type with the solar pathlength corrector, no max_sza:

For the effective_solar_pathlength_corrected I don't think the reduction is needed and setting max_sza: !!null should be fine (as your plots also confirm). For sunz_corrected I thinks it's useful, or even needed, given that the correction breaks around 90 degrees otherwise. You can see a comparison here: #2955 (comment).

So perhaps you can add your local effective_solar_pathlength_corrected definition in fci.yaml?

Thanks for the addition day/night blends. Doesn't seem to be much affected. But I believe that with these changes we can do the day/night blending at higher sunz angles to retain more of the daytime imagery, but that might be for another PR.

Thank you for your work and analysis!
I have mixed feelings about this: while I fully agree that trying to include more illuminated daytime pixels is beneficial, both RGBs do not really perform well in that twilight area... In the Cloud Type, the red-toned noise and ESL signal from the NIR1.3 is quite enhanced, and in the Cloud Phase the entire area is dominated by the blue component, making it not too useful.
It seems however that applying the default max_sza removes some of the low-quality and not-so-usable area, while still including a good amout of extra daytime pixels. So maybe that's a good compromise? If I understand the discussion correctly, the current PR code modifications reflects this setup.

There is a tradeoff between æsthetics and the ability to extract information. This is probably true for any RGB based on solar channels and also when considering the day/night-transition in geo colour. And any RGB, as well as quantitative products, will lose quality approaching the terminator. Maybe cloud type is more affected here because NIR 1.3 is so dark already.

For our operational users, the ability to extract information is more important than æsthetics. I think they will accept and understand that. Colours are a bit different and noise becomes visible. OK.

If I understand the discussion correctly, the current PR code modifications reflects this setup.

Yes, the current PR code modification uses the default solar_pathlength_corrected (correction_limit=88, max_sza=95). It's a compromise between the status quo (sunz_corrected with or without sunz_reducer) and the DWD version (correction_limit=88, max_sza=None). The compromise looks prettier near the terminator, but the DWD version has more extractable information. What do we want as default in satpy?

I agree that information content is more important than aesthetics, at least for the default recipe (a user interested in only PR-like nice looking images can still tune a different recipe to their wishes, but I'd say that the general focus should be on meteorological usability and usefulness).
My question is if the low twilight area, with all the artefacts like noise and non-representative color tones (that are at that point not describing the actual physical status of the surface/cloud scene, like the violet terminator of the cloud phase and the saturated red clouds in cloud type) is useful/delivering information to the observer. Admittedly, it likely still delivers information at least for spatial patterns, even if the spectral information is likely not very usable... so I don't have a strong opinion, I'm ok with both solutions in the end.
But at least we can leave a note in the yaml saying that using the default max_sza can help to remove some of the bad quality areas, if any user investigates this in the future.

Really nice work!
The discussion re whether to remove the terminator artefacts or not is interesting. Lets no forget though the regions where twilight lasts all day :) for these, I’m suspecting that having data with artefacts can still be better than no data at all...

The discussion re whether to remove the terminator artefacts or not is interesting.

I'm all for removing artefacts, but removing the entire area that has artefacts in it has something of throwing out the child with the dirty bathwater...

See also #3096 where @strandgren has made an excellent write-up of the different correction reductions near the terminator.

Sticking with EffectiveSolarPathlengthCorrector with max_sza: !!null, varying only correction_limit (no Rayleigh correction applied to any channel).

correction limit 86 degree:

correction limit 87 degree:

correction limit 88 degree:

correction limit 89 degree:

correction limit 90 degree:

with night_ir105:

correction limit 86 degree:

correction limit 87 degree:

differences start to become small here, because the right part is taking over when using lim_low: 78 and lim_high: 88, but of course this could be set tighter.

The same for cloud phase:

correction limit 86 degree:

correction limit 87 degree:

correction limit 88 degree:

@gerritholl Can you please add an example with this configuration as well:

correction_limit: 150
max_sza: null

which effectively disables the reduction functionality starting at correction_limit and thus falls back on the raw Li and Shibata parameterization for the full sunz range.

Sorry for my unavailability this week, ping me on slack if you want me join any discussion on this.

This is with 150 degree:

Seems to be less visible near the terminator compared to 90 degree.

This is with 150 degree:

Seems to be less visible near the terminator compared to 90 degree.

Thanks, actually I prefer this one. With 90 degrees there is a red tint for the space part. Furthermore, the ice clouds in the north are less saturated with 150 degrees compared to 90 degrees. I also thinks it an added value if we can say that we use the Li and Shibata parameterization as is, rather than a tweaked version of it.