Add compatibility with Java 9

[Don-vip]

Java 9 is well advanced in its development, early builds are now available. ​Jigsaw project is now merged into main tree.

I have added basic Java 9 support in r8777:8778, r8793, r8795, r8796:8797 but making JOSM work with Jigsaw is another challenge.

This ticket aims to list all problems encountered.

1. xmltask does not work ==> cannot build OSX package ==> ​fork created
2. ​Proguard does not work ==> cannot build optimized jar
3. ​Groovy does not work ==> cannot run some tests + cannot build taginfo files ==> fixed
4. ResourceBundle.getBundle("sun.awt.resources.awt") results in an internal error ==> bug ​reported to Oracle ==> fixed
5. Findbugs does not recognize new version string ==> ​bug and PR created ==> fixed
6. ​JEP 272 removes com.apple.eawt package ==> need to update PlatformHookOsx ==> fixed
7. ​javabug:6850612 deprecates Class.newInstance ==> replace it by Class.getConstructor().newInstance() ==> fixed
8. ​javabug:8154801 deprecates java.util.Observer / java.util.Observable ==> replace it by javax.swing.ChangeListener ==> fixed
9. Ant does not work anymore ==> ​Bug created + workaround done in r10259 ==> Fixed in Ant 1.9.8 / 1.10.0
10. ​javabug:8152912 deprecates XMLReaderFactory ==> fixed
11. Groovy bug: ​https://issues.apache.org/jira/browse/GROOVY-7879 ==> fixed
12. ​javabug:8143077 deprecates InputEvent.*_MASK constants in favor of InputEvent.*_DOWN_MASK ==> fixed
13. The workaround to disable "Internal proprietary API" warning does not work anymore ==> no way to disable it, but hidden in Jenkins builds

Unit tests:

1. jacoco has problems with nashorn: Method jdk.dynalink.linker.LinkerServices.$jacocoInit()[Z must be InterfaceMethodref constant ​log ==> See #13150 for fix
2. EqualsVerifier bugs: ​https://github.com/jqno/equalsverifier/issues/152 + ​https://github.com/jqno/equalsverifier/issues/169 ==> fixed
3. Projections off by the last digit. ==> fixed (see #11889, #13387)

Plugins:

1. Geotools does not work ==> ​https://osgeo-org.atlassian.net/browse/GEOT-5289 ==> forked

See #15560 for Java 10 compatibility.

[Don-vip]

In 8816/josm:

see #11924 - JDK9 Jigsaw: workaround for ​https://bugs.openjdk.java.net/browse/JI-9025152

[Don-vip]

In 8817/josm:

see #11924 - JDK9 Jigsaw: do not run code in headless environment (for jenkins unit tests)

[Don-vip]

In 8826/josm:

see #11924 - update xmltask to a new modern version built from ​https://github.com/don-vip/xmltask

[Don-vip]

In 8843/josm:

see #11924 - allow openjdk-9-jre in Debian control file

[Don-vip]

All plugins load fine with Java 9 except ImportImagePlugin, which relies on geotools:

23:45:31.948 [main] FATAL org.openstreetmap.josm.plugins.ImportImagePlugin.ImportImagePlugin - Error while loading plugin
java.lang.IllegalArgumentException: org.opengis.referencing.datum.DatumFactory is not an ImageIO SPI class
	at javax.imageio.spi.ServiceRegistry.checkClassAllowed(ServiceRegistry.java:733) ~[?:?]
	at javax.imageio.spi.ServiceRegistry.<init>(ServiceRegistry.java:138) ~[?:?]
	at org.geotools.factory.FactoryRegistry.<init>(FactoryRegistry.java:155) ~[geotools.jar:?]
	at org.geotools.factory.FactoryRegistry.<init>(FactoryRegistry.java:146) ~[geotools.jar:?]
	at org.geotools.factory.FactoryCreator.<init>(FactoryCreator.java:82) ~[geotools.jar:?]
	at org.geotools.referencing.ReferencingFactoryFinder.getServiceRegistry(ReferencingFactoryFinder.java:115) ~[geotools.jar:?]
	at org.geotools.referencing.ReferencingFactoryFinder.getFactories(ReferencingFactoryFinder.java:180) ~[geotools.jar:?]
	at org.geotools.referencing.ReferencingFactoryFinder.getCRSAuthorityFactories(ReferencingFactoryFinder.java:455) ~[geotools.jar:?]
	at org.geotools.referencing.DefaultAuthorityFactory.getSupportedCodes(DefaultAuthorityFactory.java:125) ~[geotools.jar:?]
	at org.geotools.referencing.CRS.getSupportedCodes(CRS.java:396) ~[geotools.jar:?]
	at org.openstreetmap.josm.plugins.ImportImagePlugin.PluginOperations.loadCRSData(PluginOperations.java:366) ~[ImportImagePlugin.jar:?]
	at org.openstreetmap.josm.plugins.ImportImagePlugin.ImportImagePlugin.<init>(ImportImagePlugin.java:88) [ImportImagePlugin.jar:?]

I have no idea what's the problem yet. Maybe linked to ​JEP 262.

[Don-vip]

Found it, it's a change of behaviour from ​javabug:8068749:

Java SE has been changed so that javax.imageio.spi.ServiceRegistry no longer allows service providers of other than Image I/O service types to be loaded. This is a behavioral change. Existing application binaries that attempt to use ServiceRegistry for non-Image-I/O services will cause IllegalArgumentException to be thrown at runtime. Such applications should be migrated to use java.util.ServiceLoader instead of javax.imageio.spi.ServiceRegistry.

[Don-vip]

Bug reported to GeoTools: ​https://osgeo-org.atlassian.net/browse/GEOT-5289

[Don-vip]

In 9199/josm:

see #11924 - add compatibility with new Java 9 version scheme (MAJOR.MINOR.SECURITY, see JEP 223)

[Don-vip]

In 9202/josm:

see #11924 - add robustness for short versions "9" and "9-ea"

[mdk]

3. ​Groovy does not work ==> cannot run some tests + cannot build taginfo files

As far as I could see, this problem is fixed in Groovy v2.4.6.

[mdk]

4. ResourceBundle.getBundle("sun.awt.resources.awt") results in an internal error ==> bug ​reported to Oracle

Also this problem seems to be fixed since 2015-11-10 (see ​https://bugs.openjdk.java.net/browse/JDK-8136804)

[Don-vip]

Yes, The Oracle bug is fixed in latest JDK9/Jigsaw builds. However Groovy 2.4.6 is not released yet.

[Don-vip]

In 9330/josm:

see #11924 - update to Groovy 2.4.6-SNAPSHOT (rev 23ad731c49dceda51e5ff324d1cfed48d7d65c49 - 2016-01-05) downloaded from CI server, to get fixes for:

• ​https://issues.apache.org/jira/browse/GROOVY-7587
• ​https://issues.apache.org/jira/browse/GROOVY-7716

There's no schedule about 2.4.6 official release, and authors say they "have only very little time these days to actually do the change", see ​https://issues.apache.org/jira/browse/GROOVY-7291?focusedCommentId=15083011&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15083011

The jar has been downloaded from ​http://ci.groovy-lang.org/repository/download/Groovy_Jdk7Build/30993:id/target/distributions/apache-groovy-binary-2.4.6-SNAPSHOT.zip%21/groovy-2.4.6-SNAPSHOT/embeddable/groovy-all-2.4.6-SNAPSHOT.jar

[Don-vip]

In 9332/josm:

see #11924 - if available, use java.runtime.version instead of java.version in status report to get build number (useful for Early Access builds)

[Don-vip]

In 9333/josm:

see #12186, #11924 - update projection regression test file for Java 9

generated on Windows with:

java version "9-ea"
Java(TM) SE Runtime Environment (build 9-ea+99-2015-12-23-183325.javare.4146.nc)
Java HotSpot(TM) 64-Bit Server VM (build 9-ea+99-2015-12-23-183325.javare.4146.nc, mixed mode)

[Don-vip]

In 9344/josm:

see #11924 - fix one javadoc warning with JDK9, add more unit tests

[Don-vip]

In 9599/josm:

see #11924 - add assertions to check if java.locale.providers enables CLDR as default because of JEP 252 in Java 9 (unit tests depending on default Java date formatting are failing)

[Don-vip]

In 9605/josm:

see #11924 - use ISO date format in unit tests to avoid differences between JDK7/8 and 9

[Don-vip]

In 9908/josm:

see #12186, #11924 - update projection regression test file for Java 9

generated on Windows with:

java version "9-ea"
Java(TM) SE Runtime Environment (build 9-ea+107-2016-02-24-182456.javare.4520.nc)
Java HotSpot(TM) 64-Bit Server VM (build 9-ea+107-2016-02-24-182456.javare.4520.nc, mixed mode)

[Don-vip]

In 10048/josm:

see #11924 - add -XaddExports:java.base/sun.security.x509=ALL-UNNAMED JVM option for JDK9 tests (see ​http://mreinhold.org/blog/jigsaw-module-system). Requires Ant 1.9.1 or later

[Don-vip]

In 10049/josm:

see #11924 - remove tab

[Don-vip]

In 10052/josm:

see #11924 - reduce double precision in unit tests to avoid tiny differences with JDK9

[Don-vip]

In 10056/josm:

see #11924 - use correct "if" syntax

[Don-vip]

In 10057/josm:

see #11924 - add -XaddExports:java.base/sun.security.util=ALL-UNNAMED

[Don-vip]

In 10058/josm:

see #11924 - Package.getPackage is deprecated in Java 9

[Don-vip]

In 10059/josm:

see #11924 - write proper html code to see it it resolves unit test failing only with jdk9

[Don-vip]

In 10060/josm:

see #11924 - remove deprecated code, reduce double precision in unit tests to avoid tiny differences with JDK9

[Don-vip]

In 10061/josm:

see #11924 - reduce double precision in unit tests to avoid tiny differences with JDK9

[Don-vip]

In 10064/josm:

see #12652, see #11924 - re-add a unit test in BugReportExceptionHandler, empty test class causes an error with Java 9

[Don-vip]

In 10065/josm:

see #11924 - rework projection regression test for java 9

[Don-vip]

In 10178/josm:

see #11924 - add compatibility with JEP 272 for Java 9 on OSX

[Don-vip]

In 10208/josm:

see #11924 - Java 9 - JDK-6850612 deprecates Class.newInstance() ==> replace it by Class.getConstructor().newInstance()

[Don-vip]

In 10209/josm:

see #11924 - Java 9 - fixes more deprecation warnings

[Don-vip]

In 10210/josm:

see #11924 - Java 9 - replace calls to deprecated classes java.util.Observable / java.util.Observer by a new class ChangeNotifier + swing's ChangeListener

[Don-vip]

In 10257/josm:

see #11924 - test possible workaround for ​https://bz.apache.org/bugzilla/show_bug.cgi?id=59556

[Don-vip]

In 10258/josm:

see #11924 - correct workaround for ​https://bz.apache.org/bugzilla/show_bug.cgi?id=59556

[Don-vip]

In 10259/josm:

see #11924 - JVM args ignored when same JVM is used

[Don-vip]

In 10260/josm:

see #11924 - use the new -XaddExports syntax specified in JEP 261

[Don-vip]

In 10262/josm:

see #11924 - add -XaddExports to access javax.xml.bind.DatatypeConverter used in JMapViewer

[Don-vip]

In 10264/josm:

see #11924 - add -addmods to access javax.xml.bind module used in JMapViewer

[Don-vip]

In 10265/josm:

see #11924 - use correct syntax

[Don-vip]

In 10267/josm:

see #11924 - use correct module

[Don-vip]

In 10269/josm:

see #11924 - remove unneeded addmods option

[Don-vip]

In 10270/josm:

see #11924 - update to Jacoco 0.7.7-20160523.090959-28 snapshot to add java 9 bytecode compatibility

[Don-vip]

In 10276/josm:

see #11924 - should be the minimal and correct addmods option

[Don-vip]

In 10289/josm:

see #12881, see #11924 - force generics to avoid Java 9 compilation error

[Don-vip]

In 10348/josm:

see #11924 - update to FindBugs 3.1.0_preview1 to add Java 9 support

[Don-vip]

In 10425/josm:

see #11924 - update to FindBugs 3.1.0_preview2

[Don-vip]

Core works relatively fine with Java 9. I have started to look at plugins, as expected they don't even compile. First error lies in Apache Commons code: ​https://issues.apache.org/jira/browse/COLLECTIONS-593

[michael2402]

I found some more problems in the unit tests.

[Don-vip]

In 10546/josm:

fix #13150, see #11924 - exclude nashorn classes from jacoco coverage to avoid lot of unit test failures (patch by michael2402)

[Don-vip]

In 10631/josm:

see #11924 - use jacoco-0.7.8-20160724.185238-10 snapshot to include ​https://github.com/jacoco/jacoco/pull/428 to fix failing unit tests since ​https://bugs.openjdk.java.net/browse/JDK-8145148

[Don-vip]

In 10741/josm:

see #11924 - XMLReaderFactory has been deprecated in Java 9, see ​https://bugs.openjdk.java.net/browse/JDK-8152912

[Don-vip]

In 10746/josm:

see #11390, see #11924 - replace javax.xml.bind.DatatypeConverter (not visible by default in Java 9) by java.util.Base64 (new in Java 8)

[Don-vip]

In 10821/josm:

see #11924 - update to jacoco-0.7.8-20160816.021912-12 (contains fix for ​https://github.com/jacoco/jacoco/pull/434 required for Java 9)

[Don-vip]

In 10836/josm:

see #11924 - fix javadoc warnings seen with Java 9

[Don-vip]

In 10837/josm:

see #11924 - extract MapCSS conditions to new class ConditionFactory (on the same model than ExpressionFactory) - should workaround Groovy bug with Java 9 (​https://issues.apache.org/jira/browse/GROOVY-7879 ?)

[Don-vip]

In 10925/josm:

see #11924 - fix javadoc warnings seen with jdk9b133

[Don-vip]

In 10956/josm:

see #11924 - suppress some equalsverifier warnings to workaround ​https://github.com/jqno/equalsverifier/issues/152

[Don-vip]

In 11001/josm:

see #11924 - use new JEP 293 options introduced in JDK9b135

[Don-vip]

In 11006/josm:

see #11924 - use new JEP 293 options introduced in JDK9b135

[Don-vip]

In 11443/josm:

fix #14211, see #11924 - InaccessibleObjectException with JDK 9 b148+

[Don-vip]

In 11590/josm:

see #11924 - remove workarounds for jdk9 compilation problems with diamond operator - sounds a lot like JDK-8075793 (solved in b150)

[Don-vip]

In 11596/josm:

see #11924 - see JDK-8157016: add required arguments to build javadoc on Java 9

[Don-vip]

In 11999/josm:

see #13387, see #11924 - allow a difference of 3500 ULPs for Java 9 ProjectionRegressionTest (due to Math.cos change of behaviour in JDK-8143353)

[Don-vip]

In 12003/josm:

see #13387, see #11924 - raise the allowed difference to 35000 ULPs to catch the worst case (34813 ULP). Logic to remove when JOSM switches to Java 9

[Don-vip]

In 12130/josm:

see #11924 - do not try old workaround with java 9 (pollutes console with stacktrace)

[Don-vip]

In 12131/josm:

see #11889, see #11924, see #13387 - use backported versions of Math.toDegrees/toRadians (more accurate and faster) - to revert when migrating to Java 9

[Don-vip]

In 12205/josm:

see #11924 - add Add-Exports / Add-Opens manifest entries so the corresponding command line arguments (--add-exports / --add-opens) are not needed anymore

[Don-vip]

In 12222/josm:

see #11924 - update to a patched version of equalsverifier that restores the Manifest and adds "Add-Opens: java.base/java.lang" to fix unit tests

[Don-vip]

In 12228/josm:

see #11924 - add --add-opens=java.base/java.lang=ALL-UNNAMED for unit tests until ​https://github.com/jqno/equalsverifier/issues/169 is solved

[Don-vip]

In 12229/josm:

see #11924 - fix module name

[Don-vip]

In 12230/josm:

see #11924 - add more --add-opens arguments for unit tests until ​https://github.com/jqno/equalsverifier/issues/169 is solved

[Don-vip]

In 12231/josm:

see #11924 - fix javadoc warnings

[Don-vip]

In 12232/josm:

see #11924 - fix deprecation warning

[Don-vip]

In 12234/josm:

see #11924 - see JDK-8165641: Object.finalize() is deprecated

[michael2402]

Replying to Don-vip:

In 12234/josm:

see #11924 - see JDK-8165641: Object.finalize() is deprecated

Are you sure this is not required any more? I added a ListenableWeakReference some days ago. It runs a Runnable as after the Object was destroyed. This can replace finalize() mechanism. But you need to take care to hold the reference to that reference.

Btw: I don't like the lifecycle of that class. It's just bad to have to keep a reference to ensure that it works. What do you think about removing it and instead extending JPopupMenu to be a ScrollablePopupMenu.

[Don-vip]

I couldn't trigger a call to the method. I assume the objects of this class are only finalized when we close JOSM, so the finalize method was not useful.

I agree this class could disappear.

[Don-vip]

In 12237/josm:

see #11924 - add --add-modules java.se.ee for unit tests so that PluginHandlerTestIT.testValidityOfAvailablePlugins can access JAXB - there is no similar option in manifest (at least not yet)

[Don-vip]

In 12238/josm:

see #11924 - make the JRE expiration date detection system work with Java 9

[Don-vip]

In 12241/josm:

see #11924, see #14649 - java 9 does not seem to include Dutch certificates yet, load them from /usr/share/ca-certificates/mozilla (see Debian ca-certificates package)

[Don-vip]

In 12243/josm:

see #11924 - update error-prone, manually built from ​https://github.com/google/error-prone/compare/sync-master-05/10/2017 for better java 9 support, see ​https://github.com/google/error-prone/issues/448

[Don-vip]

In 12244/josm:

see #11924 - simplify DNSName copy by using reflection

[Don-vip]

In 12262/josm:

see #11924 - exports/opens more packages for known JOSM plugins (geotools, jogl, kendzi3d)

[Don-vip]

In 12268/josm:

see #11924 - fix macOS dock icon with Java 9

[Don-vip]

Nicolai Parlog started a nice project: ​http://java9.wtf/ We could share our issues there.

[Don-vip]

In 12409/josm:

see #11924 - fix javadoc warnings new in 9-ea+173

[Don-vip]

In 12428/josm:

see #11924 - new versions of ant define ant.java.version to 9

[Don-vip]

In 12429/josm:

see #11924 - fix javadoc warnings with java 9

[Don-vip]

In 12459/josm:

see #11924 - replace add-opens by add-exports for sun.security.x509 package - see r12458

[Don-vip]

ASM 6.0 beta has finally been ​released. This was the root cause of our remaining failed unit tests. Now Maven team will be able to update their plugins depending on it (shade for example) and equalsverifier will use these new plugins.

[Don-vip]

In 12504/josm:

see #11924 - fix one deprecation warning for InputEvent.ALT_GRAPH_MASK

[Don-vip]

In 12517/josm:

see #11924 - use extended event modifiers, deprecate old methods - see ​https://bugs.openjdk.java.net/browse/JDK-8143077

[michael2402]

Replying to Don-vip:

In 12517/josm:

see #11924 - use extended event modifiers, deprecate old methods - see ​https://bugs.openjdk.java.net/browse/JDK-8143077

You can use the ListenerList class for managing lists of listeners. It provides:

• Optional weak listeners
• The ability to warn or throw an exception for duplicate listeners
• An easy fire method

[Don-vip]

In 12518/josm:

see #11924 - use ListenerList, fix javadoc

[Don-vip]

Replying to michael2402:

You can use the ListenerList class for managing lists of listeners.

Yep, thanks! :)

[Don-vip]

In 12519/josm:

see #11924 - fix 2 more warnings about extended modifiers

[Don-vip]

In 12520/josm:

see #11924 - fix remaining warnings about extended modifiers

[Don-vip]

In 12521/josm:

see #11924 - update unit tests

[Don-vip]

In 12522/josm:

see #11924 - checkstyle

[Don-vip]

In 12523/josm:

see #11924 - fix remaining warnings about extended modifiers

[Don-vip]

In 12525/josm:

see #11924 - suppress deprecation warning (must wait java 9 migration)

[Don-vip]

In 12526/josm:

see #11924 - fix last warnings about extended modifiers

[Don-vip]

ASM team has trouble to publish to Maven central as they changed their infrastructure, it's unlikely to see Maven plugins and equalsverifier updates before next month.

Java 9 will be released in September.

[Don-vip]

Interesting performance improvements in Java 9, see StyledMapRendererPerformanceTest/testCitySmall duration:

• Java 8: 170-190s
• Java 9: 32-52s

[bastiK]

In 12551/josm:

fix AbstractMapRendererPerformanceTestParent (see #11924)

set up NavigatableComponent, so something is drawn

[bastiK]

Replying to Don-vip:

Interesting performance improvements in Java 9, see StyledMapRendererPerformanceTest/testCitySmall duration:

• Java 8: 170-190s
• Java 9: 32-52s

This test wasn't drawing anything at the time. Still interesting, to see how long it takes to do nothing, and how much better Java 9 is at this.

In "real world" setups, I couldn't see any significant improvement so far.

[Don-vip]

OK thanks :) Indeed it was very strange!

Concerning ASM, they finally managed to push their update to Maven Central, hopefully Maven plugins should be updated faster.

[Don-vip]

In 12573/josm:

see #11924 - update to equalsverifier 2.3.2 (includes ASM 6.0_BETA)

[Don-vip]

And this is now fixed, the Java 9 build is finally stable \o/

[Klumbumbus]

congrats :)

[Don-vip]

In 12591/josm:

see #11924, see #15128 - add Java 9 option --add-modules java.se.ee for plugins requiring JAXB in Linux startup script

[Don-vip]

In 12640/josm:

see #11924 - update to equalsverifier 2.3.3 - improved java 9 support

[Don-vip]

ProGuard 6.0beta is (finally!) ​available.

[Don-vip]

In 13073/josm:

see #11924 - see #15560 - support jdk10+ in build.xml, update to proguard 6.0beta1 and error_prone 2.1.2

[Don-vip]

In 13135/josm:

fix #15535, see #11924 - make extrude mode work with Java9+

[Don-vip]

In 13233/josm:

see #11924 - Restart doesn't work on Java 9 WebStart. The command line includes jdk.plugin/sun.plugin2.main.WebStart as main class, but this is translated to jdk.plugin.sun.plugin2.main.WebStart and results in a (silent) ClassNotFoundException

[Don-vip]

In 13268/josm:

see #11924, see #14097 - ensure JavaScript engine will always be found with java 9

[Don-vip]

In 13491/josm:

see #11924, see #15560, see #16048 - explicitly generate HTML4 javadoc until all errors/warnings are fixed

[Don-vip]

In 13493/josm:

see #11924, see #15560, see #16048 - tt HTML tag is deprecated in HTML5: use code instead

[Don-vip]

In 13494/josm:

see #11924, see #15560, fix #16048: Generate HTML5 javadoc with Java 9/10/11

• name HTML attribute is deprecated in HTML5: use id instead

[Don-vip]

Proguard 6.0 has ​finally been released.

[Don-vip]

In 13495/josm:

see #11924, upgrade to Proguard 6.0, PMD 6.1.0, Groovy 2.4.14

[Don-vip]

In 13504/josm:

see #11924, see #15560 - fix deprecation warning with Java 10

[Don-vip]

In 14244/josm:

see #11924, see #15560, see #16047, see #16682 - fixes to make work proguard (OK) and animal_sniffer (still KO) with Java 9+

[Don-vip]

In 14245/josm:

see #11924, see #15560, see #16047, see #16682 - fixes to make work animal_sniffer (OK) with Java 9+