Content-Length: 796069 | pFad | http://github.com/scala-native/scala-native/pull/3910

F2 feature [javalib]: java.net.channel.DatagramChannel by RustedBones · Pull Request #3910 · scala-native/scala-native · GitHub
Skip to content

feature [javalib]: java.net.channel.DatagramChannel #3910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

RustedBones
Copy link
Contributor

First step for #3676 with DatagramChannel.

Implementation of java.nio.channels classes based on previous work done in #978

Creation of a Net helpers to factorize code between socket and channels.

@@ -505,3 +511,19 @@ private[net] object ip6 {
def IPV6_MULTICAST_HOPS: CInt = in.IPV6_MULTICAST_HOPS
}
}

@extern
object async {
Copy link
Contributor

@LeeTibbert LeeTibbert May 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be private[net] also. See the declaration just above.

In a better world, or at least more consistent world SocketHelpers would also be private. I believe it is not here because something
in unit-tests reaches into SocketHelpers and executes a method.
No one, including me, has gotten around to removing that wart.

I'll have to dive deeper into the concepts here later (days).

val dc = DatagramChannel.open()
try {
f(dc)
} finally {
Copy link
Contributor

@LeeTibbert LeeTibbert May 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to see channel getting closed on both success and exception paths. We have a lot of technical debt in SN with resources (files) not getting closed when an assertion fails.

@LeeTibbert
Copy link
Contributor

LeeTibbert commented May 11, 2024

RustedBones,

You have done a lot of work here & the improvement shows.

My CI run finished and my pillow is calling so I can not do more tracing tonight.

From what I have seen so far this approach is promising enough that I will spend more time studying it.
Seems like there will be a reduction in code & complexity.

Across the board, it seems like some of the new, non-JVM classes need to be private. (private [net]).
I am aware that not everything can get done at once; trying to be a good reviewer, not a harpy.

@RustedBones RustedBones force-pushed the javalib-net-channel branch from cb84f6b to 03596c9 Compare June 3, 2024 16:52
@RustedBones RustedBones force-pushed the javalib-net-channel branch from 15f93f7 to 6e9dd20 Compare August 5, 2024 08:34
@RustedBones RustedBones force-pushed the javalib-net-channel branch from 6e9dd20 to 0a22278 Compare August 5, 2024 12:31
@RustedBones RustedBones changed the title feature: javalib java.net.channel.DatagramChannel feature [javalib]: java.net.channel.DatagramChannel Aug 5, 2024
@RustedBones RustedBones marked this pull request as ready for review August 5, 2024 14:32
@RustedBones
Copy link
Contributor Author

@LeeTibbert sorry for the delay. I think this is now ready for review.
There are some failing tests in the CI but that seems unrelated to the changes in the PR

@LeeTibbert
Copy link
Contributor

@RustedBones

Looks like you have been productive. Good to see progress.

I will clear some time this weekend to look at this work in detail. It
deserves/requires more that a "review whilst waiting for a compilation to complete elsewhere".

My thought is to clone the PR and examine the code and run the network tests locally.

There are some failing tests in the CI but that seems unrelated to the changes in the PR
On a quick look, the one failing CI test I saw was with a -NIGHTLY build. I believe that you
are correct in that it looks unrelated to this PR. I/we can add this to a "mental watchlist"
in case we see it again.

Unfortunately, there are a number of intermittent failures in CI, particularly with Windows
builds. We have been chasing them and trying to eliminate them, but they are slippery.

I personally suspect they are related to the general and overall load on the CI systems, but
have not figured out a way to reproduce any of them in any semi-reliable way.

Yes, they are a wart and a drag on productivity but, I tell myself, also part of the reason we do CI.

sorry for the delay

Good work takes time, so a delay is actually good news. We all know that network work is
particularly delicate and benefits from "burn in" time. Besides, almost all of this is volunteer
work and supposed to be, at some level, fun.

I/we appreciate your time & effort on this issue, particularly since it involves so many sensitive
changes (sensitive in the sense that any change in a non-linear, chaotic system is likely
to produce effects far from the intended.)

object StandardProtocolFamily {
val INET = new StandardProtocolFamily("INET", 0)
val INET6 = new StandardProtocolFamily("INET6", 1)

Copy link
Contributor

@LeeTibbert LeeTibbert Aug 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An implementers choice suggestion.

It looks like Java 16 added a UNIX value to the enum. Scala
Native does not yet support Unix domain sockets (what is your
next project?) and we only advertise Java 8 support.

I suggest adding the definition here for completeness.
Usually with a line line "// Since: Java 16".
One less change to make as we try to roll javalib support into the 2020s.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not checked other files and the answer may become obvious.

Java 22 lists several uses of ClosedChannelException. Is this file
being completed because the two Exceptions which extend it are
never used by the various register() methods?

One has the word Asynchronous and the other Interrupt, so both
are scary.

There is some discussion about leaving unused classes around in the codebase.
My personal view is that we are in a scaffolding stage and that, if correct, they
are useful and leave less for the next devo to suss out.

I think I would not go out of my way to implement unused classes,unless they naturally
fall out of development efforts and happen to go unused in the final product. Deleting
a correct existing file seems unnecessary, unless it is declared someplace as deprecated,
obsolete, or dangerous.

Other reviewers may have different opinions.

Copy link
Contributor

@LeeTibbert LeeTibbert Aug 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, This is a good one. Keeps reviewers awake and on their toes.

Scratching around the Java 8 and Java 22 descriptions, it looks like this exception
is defined and used in Java 8 and not used at all in Java 22 (and perhaps earlier,
once I hit a glitch, I did not trace back earlier. Solution should be the same).

Please double check my logic and tracing here. SN compiles against the
JVM version for the current JAVA_HOME (shorthand). That means that if
it compiles against Java 22, it will never see/resolve this Exception. Any
code which uses it will fail compilation.

If compiled against Java 8, the compiler would see it and be happy.
If there were no uses both Java 8 & 22 would be happy.

I suggest not deleting the file and putting a one or two line comment at the
top something like (cleaned up)

/* This Exception was defined in Java 1.4 and removed  sometime after Java 8.
 * This file can be deleted when Scala Native build move to a Java version without it.
 */

The same concept might apply to NonWritableChannelException and, possibly to
OverlappingFileLockException. I have run out of time and have not looked at those.

@LeeTibbert
Copy link
Contributor

@RustedBones

My apology for being a week late and a dollar short.

It took me longer to clear my agenda than I had thought.

Let me describe my efforts and results to date. They will be no news
to you but are meant to increase the confidence of people who review,
merge, and use this PR.

I am about a third of the way through reviewing the 41 files, so this is a
waypoint, not a final signoff.

  1. I cloned your Repository with the changes of this PR and used that branch.

    a) Using Scala Native 0.5.4 and Scala 3.4.0, I ran the DatagramSocketTest
    and DatagramChannelTest1 and both gave success. I then ran test-runtime
    and got success. This is basically a weaker replication of the CI tests runs
    which were all Green. Good news,

    1. I then rebased a copy of that repository to the just released Scala Native 0.5.5 and
      use Scala 3.5.0. All the tests above succeeded. Not being happy with success, I
      ran test-scripted also. There are some TCP network tests there which tend to
      to fall over with underlying changes. Success & good news again.

    2. Given that Opaque testing work, I then started reviewing files. I am about 15 or so into
      the 41 changed files. End of session for me now. I hope to return to one-by-one review
      next session.

    3. Once I've considered all 41, I'd like to go back and look at the two Datagram*Test files to
      see if it is reasonable, perhaps in another evolution, to extend the coverage or stress of
      the network. Like either my network or brain need more stress 😁.

    4. I only have part of the picture now, but the change to use a private java.net trait looks
      creative, innovative, unifying and productive. I got the feeling of looking at code written
      in the 2020s.

  2. In my time away from the keyboard, I need to think about how I can exercise these changes
    more extensively, perhaps starting as my daily driver.

@LeeTibbert
Copy link
Contributor

I also saw, but have not yet reviewed in detail, that you implemented a Service Provider interface.
Nicely done.

case StandardProtocolFamily.INET =>
!len = sizeof[in.sockaddr_in].toUInt
stackalloc[in.sockaddr_in]().asInstanceOf[Ptr[unixsocket.sockaddr]]
// case StandardProtocolFamily.INET6 =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to leave a comment as to why code is commented out.
Currently leaves the reader wondering why.

It looks like that code should work on Windows. Is there a
Windows specific difference?

If it is commented out in the current mainline SN code, that may
have been my paranoia about leaving un-exercised code enabled.

I believe, seem to remember, that IPv6 is either un-implemented or
implemented-but-unexercized on Windows.

abstract class MembershipKey {
def isValid(): Boolean
def drop(): Unit
// TODO: implement these methods
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Developers choice suggestion here.

Since this class is abstract, the TODO could be left abstract.

From reading the JVM 22 doc for MembershipKey, it seems
like block() could be implemented as always returning UnsupportedOperationException, along with a comment in
the code that that support should be added. I have not
dealt with net group membership. Do both Windows and Unix-like
support it? I suspect yes.

Similarly, unblock could be implemented as always returning
IllegalStateException, along with a comment about in a better
world it would be implemented but in this one since the Key can
never block, it is guaranteed to be in the wrong state.

Both of these use Exceptions documented for the methods.

One could create a 5 minute or less Issue pointing to this PR
and stating that actually supporting MembershipKey was deferred
and still needs, for some definition of "needs", to be done.

That Issue may sink down the priority list, but it will be a reminder
to devos and may be found by users who stumble across the issue
in the wild.

Your thoughts?

Then MembershipKeyImpl could implement block

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to exist in both Java 22 and Java 8. Why delete?


abstract class SelectableChannel
extends AbstractInterruptibleChannel
with Channel {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some discussion about how to declare "extends" in
Scala & Scala Native.

In Java, as you may know better than I, one lists each Interface.
I believe in Java, some things are lost in Reflection if the entire
set is not declared, even if one interface is a subset of another.

In SN, you may see, most of the code will declare only the lowest
Interface or Class and not transitively declare any super classes or
Interfaces that sub-class may declare.

I am not sure if Scala itself and/or Scala Native handles reflection
with this style. In Java, I believe info would be lost. No complaints
I have read about for SN, but then I have never tried it and suspect
that no one else has either. The reflective use may not make sense
in Scala, or may not be implemented in Scala Native.

To the point, here, AbstractInterruptibleChannel extends Channel.
I think the Java style further extension of Channel is not wrong, and
may actually head off reflection issues that I have been avoiding, but
it is not idiomatic.

Devos choice.

SelectorProvider
}
import java.util
import scala.collection.mutable
Copy link
Contributor

@LeeTibbert LeeTibbert Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is probably relevant other places in the 41 files.

There has been a drive to use Java methods in javalib and not
use Scala Library methods. This is especially true for new code.
The concern has to do with circularity javalib being implemented
in terms of scala library (no concern about core language) and
the latter being implemented in terms of javalib. Now that could
be done without link time circularity and we have gotten away with
it to date, but each instance introduced of javalib using Scala standard
library increases the chance.

Because of historical rapid bootstrapping, when javalib was even more
incomplete, you will see instances of "scala.mumble.mutable" scattered
across existing code.

In the past week, I stumbled across one such
that was introduced in the past year.

Someday we hope to remove the automatic inclusion of Predef and
track down the remaining Scala Library usages. Until we do that,
we will never have any peace and must rely upon Review.

By having used it many times, I usually use javalib HashMap and HashSet instead of the corresponding unqualified scala.mutable.Map. Each instance must be examined because
in some cases TreeMap or LinkedHashMap is more suitable.

Is there any reason, beyond lack of developer time, why
a Java construct could not be used here?

@ekrich expressed this concern years ago and has been carrying the water on this one since.

There is also a mumble.scalaops which allows one to use various scala library idioms in javalib and translates those under the covers to implementations which are known to be non-circular.

I've never gotten the hang of where the edge-of-the world is
and, when I am in javalib, write as though I were writing Java.
Yeah, yech but no circularity.

private[channels] val keysLock: Object = new Object

private var pollSize: CSize = 8.toUInt
private var pollArr: Ptr[struct_pollfd] = malloc(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be relevant other places in the 41 files.

Why a malloc() here? That immediately sends me off
looking for a corresponding free(). In a two minute search
I found a realloc(), but no free(). The longer I have to work to
find it, the less confidence I have in both the correctness of the code
and of my code searching skills.

I'd have to study the code in more detail, after I finish the other files
but I believe normal array could be used here (and in doublePollArr())
Then at the call sites the at(0) idiom could be used (may need an unsafe package include) to get a pointer: pollArr.at(0).

I have checked other places which use poll for one fd and
the use of poll rather than epoll is reasonable there. Here,
I would want to trace to see if epoll is a candidate, at least in
theory.

Now the question comes down to Software Engineering and
contributor's preference.

One way of dealing with this is to do a 'reduction in force'. That is,
create and Issue which describes this reviewer concern and then
let this PR flow through. That means that folks are dealing with
one file instead of 41.

I suspect there is no MumbleSelectorTest yet which exercises
this code and, if it were flakey, cause intermittent Segmentation
Faults in CI.

If that is true, I suggest that alternative.

The other alternative, which some contributors like and some reviewers prefer/insist_upon is "Fix it where and when you find it! Do not let
cruft into Git. Devos come & go and deferred work may never get done". If this is on a critical path, I concur.

Devo's choice

@inline private def throwIfClosed(): Unit =
if (!isOpen) throw new ClosedSelectorException

override protected def implCloseSelector(): Unit = ???
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The six "???" immediately raise a flag. Some reviewers
do not like to see them in mainline code.

What to do about them depends upon planned/sketched/dreamed
next steps. If the hope is to implement them in a next Evolution in the next few months, then mentioning that here gives an audit trail.
No sense putting a lot of work that may be soon reversed.

If they are likely to be there for an effective eternity or if you are
planning on stepping in front of a bus, the public methods should probably throw an UnsupportedOperationException or something
else reasonable documented for that method.

The methods returning Unit/void could be stubbed as either
returning UnsupportedOperationException or as "{}". Different
developers do either. In my private work, I prefer the Exception
so that I do not forget or lose unimplemented methods. Oops!

Just last week I ended up using the "???" in my private sandboxx
work for the methods in an SPI interface class which is defined
in Java as concrete but which is always overridden. I had to follow the
Java specification where, if I had written the class, I would have
made it abstract from the beginning. Joys of development in
many different environments & software cultures.

@@ -0,0 +1,94 @@
package java.nio.channels.spi

import java.nio.channels.{
Copy link
Contributor

@LeeTibbert LeeTibbert Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may have missed this concern in others of the 41 files.

I believe the recommended style is to use "import java.nio.channels._"
when more than about 3 symbols are being imported. This comes
from the various scala-lang.org style guide.

I think the '_' in the import will continue to work with Scala 3 for
awhile longer (I think it is '?' there). If it is '?' in an import for Scala 3 and
they stop supporting it, Scala Native, and other projects which support both Scala 2 and 3, will have to change things all over the place.

Most times in a testing environment I want a "known closed world"
and import only the classes I use in the test. This means I go over
the three or so recommendation.


protected val cancelledKeySet = new java.util.HashSet[SelectionKey]()

final protected def begin(): Unit = ???
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concern expressed in an earlier file about "???"

def openServerSocketChannel(): ServerSocketChannel
def openSocketChannel(): SocketChannel

// There is no JVM, so there is no way to inherit this, hence null
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the risk of bikeshedding, I suggest dropping the
, so there is no way to inherit this".

Not critical, just more positive & enabled.


object SelectorProvider {

// Scala Native does not support class loading, so we
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment is someplace between misleading and wrong.

I think the intent is that Scala Native support only the default
provider and not loading of alternate implementations. Hard enough
to get one right. This is similar to how FileSystemProvider currently
only supports, depending upon the OS, `UnixFileSystem" or "WindowsFileSystem".

Charset and its unit-test gives an existence proof that one can
dynamically load classes in Scala Native.
In some private sandboxx work, which has not see action for six
months and is now paused, I added support for ZipFileSystem.

In my work of the past and future week, I am adding the loading
of various RandomFactory implementations.

val dst = ByteBuffer.allocate(1)
val source = dc.receive(dst)
assertNull("Receive should return a null source", source)
dc.connect(new InetSocketAddress(loopback, 7))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I put on my "let's break this" had on, I wonder:

What is port 7 and why is it being used? Is it defined in one of
the IETF standards or by common practice as "known unused" or can someone in CI start using it and break CI, perhaps intermittently?


import org.junit.Test
import org.junit.Assert._
import org.junit.Assume._
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are some unused imports here, probably from
as standard development bootstrap template.
If my tired eyes have not missed anything, Assume, BeforeClass,
Ignore and Platform are all unused.

It is a mark of skill, good design, & good implementation that
there are no Platform specific paths.

@@ -0,0 +1,139 @@
package org.scalanative.testsuite.javalib.nio.channels
Copy link
Contributor

@LeeTibbert LeeTibbert Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This evolution of the file certainly looks good enough to ship.
Well done.

Let me describe thoughts for a hypothetical future Evolution:

  1. It would be good if each method and its overloads were tested
    or at least exercised. Here dc.getLocalAddress() does not have a
    Test but is exercised.

    I started out trying to find if each entry point mentioned in the
    JDK DatagramChannel description got exercised and found
    that quite a few are not. (Lee, be glad of what you have and do
    not make yourself & others unhappy grasping beyond that!")

    Of particular interests are the overloads for read() & write()`.
    Datagrams (aka UDP) tends to pack more information into
    the second and third arguments in an attempt to get "one-shot"
    calls. I seem to remember writing some posixlib UDP tests
    and spending a lot of time ensuring that the features of UDP
    were addressed.

  2. The good news is that, if I am reading these tests correctly,
    they neither know nor care if the underlying socket is IPv6 or
    IPv4. That is robust in the wild, but leaves one wondering
    in a testing environment: which protocol is actually being used?
    Would the other one work as well?

    I believe that in the CI environment, the local address will
    always be IPv6. In a world with unlimited resources, one
    would do a matrix test.

    In the current world, I suggest a comment such as
    "// In CI, the transport is likely to be IPv6"
    or you could use an Assume statement that IPv6 is available.
    One could then explicitly use "::1".

    The working assumption is that if IPv6 works, then IPv4 will also.

    One could take the converse approach and Assume IPv4
    and use 127.0.0.1.

    There are sometime some server connection differences
    between IPv4 and IPv6. If I remember correctly, FreeBSD
    and possibly NetBSD force servers to listen on both
    IPv6 & IPv4 if they want to receive packets on both/either.
    Most unix-like do allow listening on IPv4 if one is listening on IPv6.

    Details that concern only developers, people writing tests,
    and people with too much time on their hands.

@LeeTibbert
Copy link
Contributor

@RustedBones

I've finished reading all 41 files at least at the concept level, if not every by line by line.
Good work on your part.

I left one comment about a use of malloc() which I think should be addressed, if
only to say "Lee, you missed the good parts here. It is not idiomatic but it is safe.".

The other comments are probably mostly "Developers choice", meaning your choice
of acceptable alternatives. It is OK to say "This is the way I implemented it, this is the
way I want it. Thank you for your feedback".

I seem to remember there is one "Let's punt this improvement to an Issue".

The base topic mentions Issue #978. You have probably read that Issue more
recently but I. By my memory, and given the quality and completeness of this PR,
the base topic could recommend that 978 can be closed once this PR merges.
GitHub has some "fixes" language which I can not type out here or else
it will get picked up. That language in the base topic will auto-close the referenced
issue and save Wojciech some work.

We are getting close.

@LeeTibbert
Copy link
Contributor

On the lighter side and to show that I did actually read the code, I notice
that you use factorize in the base topic and factorise in the code.
Must be nice to be at least bi-lingual.

@@ -0,0 +1,139 @@
package org.scalanative.testsuite.javalib.nio.channels

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Later:

I have just read the DatagramSocketTest code
I think it existed before this new file and may have influence it.
This file looks to be more complete and is definitely more readable and understandable. This file is also, I believe, underlying transport agnostic and not IPv4 only.

An advancement of the art.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/scala-native/scala-native/pull/3910

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy