So... oops, Cleaner is Java 9+. BUILDING.md says JDK 8 or up, so my PR should probably be targeting that.

I'm planning to reimplement it directly on top of the Reference API some time after the christmas break. It'll probably take a while because the Cleaner API makes the boundary cases much easier to handle... but obviously that is of no help where it isn't available.

@matthias-fratz-bsz Do the PR against the 10-dev branch for now... that's our JRuby 10 branch and it will require Java 21 minimum.

We'll have to evaluate options for 9.4 since that must continue to support 8 for at least a year.

Do the PR against the 10-dev branch for now.

... using the Cleaner API.

Bonus points if you want to do a separate PR for 9.4 that just emulates Cleaner with threads and references!

@headius I ported the Cleaner API version to 10-dev as PR #8561 (if "git cherry-pick" can be considered porting), and added a simple implementation here that just uses a cleanup thread per LocalContextProvider.

For simplicity, that replacement code isn't as sophisticated as Cleaner: If you forget to call terminate(), the thread sticks around forever. Cleaner does some really neat trickery to terminate its thread once the Cleaner instance itself has been garbage collected... but I don't think this is necessary here. It's a daemon thread that doesn't keep the JVM from exiting, and terminate() does always seem to be called eventually.

@matthias-fratz-bsz Thanks for the extra work! I should have been more clear... JRuby 10 will depend on Java 21, so it can use the normal JDK Cleaner. But in order to fix the issue in 9.4, you have written your own version (which is also good).

I'll have a chat with @enebo and decide how we want to approach this. It might be simplest to merge what you have here for 9.4.x, merge that forward to 10, and then do another PR at some point to move 10's finalizer logic (ALL of it) to the JDK Cleaner.

@headius The way I understood you, was to create a Cleaner-based implementation for JRuby 10. There wasn't any extra work; my original implementation works just fine when rebased onto the 10-dev branch. That's in the other PR though, because I didn't find a way to change the branches on this one. Sorry if that caused confusion.

The 9.4.x solution using a separate thread (ie. commit 71260b8 here) is a bit of an afterthought. That obviously isn't needed for JRuby 10 (which is why it isn't in the other PR), but I made it anyway when I realized that it would be fairly easy. I'd feel better if that code doesn't stay forever though, because I'm not entirely sure that its cleanup thread is properly terminated in all situations (especially in eg. Tomcat).

It's probably indeed easiest to merge this PR here into 9.4.x, merge it forward into 10-dev, and then git revert 71260b8 on the 10-dev branch. The other PR is then pointless.

Ok this is going to need more review and some discussion with @enebo @kares because it introduces another "cleaner" thread.

To clarify for anyone catching up... this PR is a 9.4 version of #8561, but with a small hand-made Cleaner since we can't use the Java 9+ version. This may be fine to merge, but introducing another thread demands some extra scrutiny.

@matthias-fratz-bsz How critical is this fix to you? We are hoping to get another quick release out (9.4.11.0) with some critical fixes, but I still have some concerns about the lifecycle of this new cleaning thread. I'd like to postpone until 9.4.12.0.

I will look it over again today and see if I can assuage my concerns.

@headius Go ahead; this is not a time-critical issue for us. The memory leak in Cantaloupe is slow enough that daily restarts, while ugly, are a serviceable workaround. Also, cantaloupe-project/cantaloupe#715 (essentially: turning off Variable Sharing) should reduce the amount of memory leaked to a point where it is essentially irrelevant.

@matthias-fratz-bsz Ok thanks for the feedback. I provided my review (with approval) so we will plan to merge this in for 9.4.12.0 (unless @enebo thinks it is critical enough to go in .11).

I also analyzed the thread's lifecycle a bit more closely:

• It gets started essentially from the constructor. This isn't exactly best practice. Maybe it would be better to delay its start to, say, the first time a LocalContext is actually requested? Might be worth the slightly more complicated code.
• It is only terminated from LocalContextProvider.termiante(). So if the LocalContextProvider were to be garbage-collected without calling termiante(), the thread would absolutely stick around forever (well, until JVM termination). However, LocalContextProvider.terminate() is called from ScriptingContainer.finalize() (via ScriptingContainer.terminate()), so there should be no situations where LocalContextProvider gets garbage-collected without having terminate() called first.

@headius I'll try to make such a test case and add it here. Current idea is to hold a PhantomReference to the thread's LocalContext we want to monitor, then keep poking the GC until the reference gets enqueued proving that the LocalContext was cleaned. I haven't tried it but I think it'll work.

@matthias-fratz-bsz that sounds like a good plan.

@headius So... third time's the charm, I guess. This testcase now checks that:

• When the test thread quits, LocalContext.remove() is called and the LocalContext is garbage-collected.
• When LocalContextProvider.terminate() is called, so is LocalContext.remove(). Once the test thread then terminates, its LocalContext is also garbage-collected.

The test for LocalContext being garbage-collected uses a PhantomReference as discussed. For checking that remove() was called, I found it easier to just override the method in a subclass. I'm checking both LocalContext.remove() and garbage-collection because LocalContext.remove() is presumably important to clean up resources, and garbage-collection avoids the memory leaks that got me here.

@matthias-fratz-bsz Thank you for the test! Looks like about as good as we can get it considering we're trying to test GC effects.

We may need to do another "quick fix" 9.2.12.0 release that doesn't include this (trying to minimize changes again), but I think it's ready to merge into master after that.

@enebo This could use some review but I believe it is ready.

Not sure if this got rebased weirdly but I've restarted the CI run to see if the fails go away. If they don't we need to figure out what's wrong before merging.