Support for JPMS (Java Platform Module System)

[Markus KARG]

Using Babbler with JDK 9 is not an easy task, since Java 9 comes with JPMS enabled -- hence, all Babbler packages are reported as missing. The problem is that people might like to migrate their projects to Java 9, so even if Babbler stays in JDK 8 there will be problems we have to solve. I tried to get it right without modifying Babbler in a first step, but it is really hell, simply spoken, so I gave up after one hour. We have to modify it and provide some help ontop.

Hence I'd like to propose a plan for Java 9 support:

Step 0 -- New git branch jpms

Step 1 -- In MANIFEST.MF we provide a new Automatic-Module-Name entry so the current Babbler JARs will still run on JRE 8 and JRE 9 without any change, but the module name will be fixed right now, so it will not change when later modularizing Babbler. Possibly we can skip this step if all JARs have names exactly reading like the wanted module names (to be checked).

Step 2 -- Provide a public example how to use Babbler in a JDK 9 project. This essentially contains a list of the fixed module names provided in Step 1 and possibly a short exemplary POM.

Step 3 -- While still compiling for JRE 8 using JDK 8 tools, replace Automatic-Module-Name by module-info.java compiled using JDK 9 tools (there is a Maven example on the Maven Compiler Plugin web site). Using this step, JRE 8 and JRE 9 will still be able to use Babbler rather unchanged, but we are not forced to build Babbler completely using JDK 9, which might take some time to get it right, apparently.

Step 4 -- Build using JDK 9 tools for target JRE 8, so we can upgrade our build machines without forcing users to upgrade their target runtimes.

Step 5 -- Build using JDK 9 tools for target JRE 9. IMHO this step cannot be done in the Babbler 1.x line, but we have to wait with this step until we start a Babbler 2.x line -- or at least, we have to start that line with this sole commit inside (and possibly more Off-Topics like support for Flows API).

[Markus KARG]

@sco0ter If you like the plan, I could spend some time and send you a PR. But it will need some weeks, as I am totally booked out in October.

[Markus KARG]

Had a deeper look into the existing structure of Babbler discovered some more problems to fix, unfortunately, so the above plan is not enough. In particular, Babbler's JARs produce split packages. This means, there are packages defined in more than one JAR, which is essentially not allowed in JPMS. All current solutions I did find either enforce the consumer to explicitly hack a lot of command line options -- or enforce to fix that. In the end, split packages never had been a good idea, even back in Java 8. So it might be better to sort out this issue in the current master branch already, before even thinking of Java 9...?

[Christian Schudt]

I didn't spend too much time with JDK 9 yet, but I thought JDK 9 would allow "old" Jars to be on the module path as "automatic module", where the module name equals the Jar name.

I am open to do a migration to JDK 9 as long as it runs on JDK 8 as well (Step 3 ensures that?).

What split packages are there? I tried to prevent them actually, because I knew they were troublesome.

[Markus KARG]

Yes, JDK 9 allows "old" JARs to be on the module path as Automatic Modules, but the name of the module then is the name of the JAR. If that fits all cases for Babbler, then we can skip Step 1.

Step 3's sole purpose is to guarantee that the JARs still will be targetting JRE 8, but additionally will work as modules on JRE 9.

Anyways, one restriction of Modules is that they must not split packages among several JARs -- which Babbler does. At least for all (!) sub-packages of rocks.xmpp.extensions, because these sub-packages are found in both, xmpp.extensions and xmpp.extensions.client. This was reported by javac (v9), and I assume that there might be more cases when manually checking it (or using jdeps).

I checked several tutorials, but I do not see any other solutions that either consolidating the JARs or renaming these packages.

[Christian Schudt]

Hm... there's the package rocks.xmpp.extensions.chatstates in the xmpp-extensions.jar and the rocks.xmpp.extensions.chatstates.model package in the xmpp-extensions-client.jar.

Is this an issue? I assumed that both packages are distinct packages. Each have their own package-info.java. I really thought they were not "split packages". In fact I read into the topic a few years ago and decided to put all "model" sub-packages in the other jar to avoid exactly this issue.

[Markus KARG]

Seems like a misunderstanding. Yes, there is no package-hierarchy in Java but the package namespace is flat. But what I actually meant was the fact that the folder xmpp-extensions/src/main/java/rocks/xmpp/extensions/ seems to contain virtually the exact same folders (e. g. filetransfer) than the folder xmpp-extensions-client/src/main/java/rocks/xmpp/extensions/. As JPMS treates both folders as separate modules, JDK 9 complains that e. g. the package rocks.extensions.filetransfer is exported by both automatic modules.

[Christian Schudt]

If you look at the folders you are right. On the other hand, why doesn't it complain, that the package rocks isn't exported by multiple modules then? In your example, rocks.extensions.filetransfer was a mistake, which should be fixed. It is indeed a true split package. But the majority of packages should be fine, like the chatstates package.

[Markus KARG]

Thank you, this clearly explains why JDK 9 complained exactly about that -and only about that- particular package! So introducing /model/ in filetransfer should definitively fix it! Great. :-) I will check that immediately...

[Markus KARG]

I started to work on this issue and will send a PR when done.

[Markus KARG]

Split package problem solved; JPMS seems to be happy now. First commit recorded in my jpms branch. Will go on tomorrow -- too tired...

[Markus KARG]

Step 0 finished. Step 1 finished. More tomorrow. Good night.

[Markus KARG]

@sco0ter Question: What JMPS module names do you want me to use? My proposed commit currently would use the following (reversed DNS names just we are used with Java package names):

• rocks.xmpp.addr

• rocks.xmpp.core

• rocks.xmpp.core.client

• rocks.xmpp.debug

• rocks.xmpp.extensions

• rocks.xmpp.extensions.client

• rocks.xmpp.fx (Maybe you like rocks.xmpp.javafx better?)

• rocks.xmpp.sample

• rocks.xmpp.websocket

A counterproposal would be using babbler.* instead of rocks.*.

What do you think?

[Christian Schudt]

These are fine. Go with the rocks.xmpp names. It's consequent with the package names then.

[Markus KARG]

Great. I will start with the example for Step 2 tomorrow.

[Markus KARG]

Step 2 done. Will file a PR now, Babbler 0.8.0 will now run fine on JRE 9 and all subsequent steps enforce using JDK 9 to build Babbler.

[Markus KARG]

Pull request #33 filed. I now will start to slightly modify the build system so JDK 9 is needed for only additional module-info.java files (these cannot be created using JDK 8 due to the new module keyword). Hence, an additional JDK 9 is needed from now on to build Babbler, but not for its usage.

[Markus KARG]

Working on Step 3. This is really hard work. Lots of exports to add... Stay tuned. :-)

(Also filed PR against Precis project, to prevent using filename-based automatic module neams)

[Markus KARG]

Work on Step 3 needs a little while, as I felt sick today. :-( Hope to finish it at the weekend.

[Markus KARG]

Still sick. :-(

[Markus KARG]

Healthy meanwhile. Step 3 is half way done. I can build using javac 9 and provide java 8 class files plus java 9 module-info. Next step is to not fail lots of tests... Stay tuned.

[Markus KARG]

On JDK 9 mvn clean test runs successful meanwhile, but mvn verify fails still -- will look into that now.

[Markus KARG]

Had to skip analyze-only as it internally uses ASM, and ASM throws IllegalArgumentException on JDK 9.

Had to skip errorprone as it fails on JDK 9.

Had to consolidate steps 3 and 4, as it is impossible to compile module-info.java on JDK 8.

@sco0ter Steps 3 and 4 are successfully finished, with some limitations. mvn clean install works fine on JDK 9 with JRE 8 + module-info.class as target. You can check the current state if you like. Further work on this issue is on hold, until ASM and errorprone are fixed to work fine on JDK 9, which I suspect is not within 2017. See https://bitbucket.org/mkarg/babbler/branch/jdk9.

[Christian Schudt]

Concerning errorprone, maybe this helps: http://errorprone.info/docs/installation (see Java 9 section) https://github.com/google/error-prone/issues/448

[Markus KARG]

Regarding errorprone: The problem is not that it fails due to technical problems, but that it simply is unable to compile against Java 9 source code (quote: "Support for using the Java 9 language level (not just running the tool on JDK 9) is work in progress.") -- It simply fails with the message "release version 9 not supported". As there isn't anything we can do here, we just have to wait for google to fix that. I do not see any benefit from skipping errorprone just for module-info.java; instead I would just suspend my prepared work and go on once google is done with JRE 9 language support.

[Markus KARG]

@sco0ter I don't dare to tell, but there is new PRECIS trouble on JDK 9 since your latest changes. While it was all fine before, today (i. e. including all your latest PRECIS and Babbler changes) testing on JDK 9 now fails:

[ERROR] testLocalPart(rocks.xmpp.addr.JidTest)  Time elapsed: 0.024 s  <<< FAILURE!
rocks.xmpp.precis.InvalidCodePointException: Invalid code point at position 0: 0x149
        at rocks.xmpp.addr.JidTest.testLocalPart(JidTest.java:387)

[Christian Schudt]

I know this issue. Unfortunately there is ongoing confusion about the correct order of rules in PRECIS.

See https://www.ietf.org/mail-archive/web/precis/current/msg01443.html

The specs are misleading. People in the mailing list tell me different things as written in the spec. I'll wait a few days and then sort this out.

You are working on two snapshots, the Jid Tests currently don't work with the head of precis trunk.

[Christian Schudt]

Everything should run again :-)

[Markus KARG]

Thanks for fixing this! Yes, no more PRECIS trouble currently. :-)