Resolve all dependencies and tools using Apache Ivy

[wiktorn]

Follow up to: #8269 and #16420. Needed for: #16858

Use JMapViewer from our Nexus instead of use of svn:externals. This allows us to:

• test in the field, how well does it's job in controlling our dependencies
• move JMapViewer outside OSM SVN server

The goal is to do incremental changes and provide better dependency management than following HEAD on svn:externals.

Sponsored by: OpenStreetMap Hack Weekend, Karlsruhe Oct'18

[simon04]

I merged the patch with the recent changes and uploaded attachment:ivy-jmapviewer-v2.patch​

Is there anything holding us back?

[stoecker]

Is there anything holding us back?

Yes. The new server. We don't want to install and do anything new on the old one.

[simon04]

Since we got a new server in the meanwhile, is anything (else) holding us back?

[Don-vip]

Replying to simon04:

is anything (else) holding us back?

I don't know yet how to complete the GitLab setup. I'll comment why in #16857 comments.

[simon04]

In 15977/josm:

see #16860, see #18140 - Setup Apache Ivy (patch by wiktorn, modified)

[simon04]

In 16011/josm:

see #16860, see #18140 - Restrict SpotBugs to org/openstreetmap/josm

[Don-vip]

In 16020/josm:

see #16860, see #18140 - better way to restrict SpotBugs analysis scope (without error about missing classes)

See ​https://github.com/spotbugs/spotbugs/blob/4.0.0/spotbugs-ant/src/main/java/edu/umd/cs/findbugs/anttask/FindBugsTask.java

[simon04]

In 16021/josm:

see #16860 - Resolve org.glassfish:javax.json via Apache Ivy

[simon04]

In 16022/josm:

see #16860 - Resolve org.tukaani:xz via Apache Ivy

[simon04]

In 16023/josm:

see #16860 - Resolve org.apache.commons:commons-compress via Apache Ivy

[simon04]

In 16024/josm:

see #16860 - Resolve commons-logging:commons-logging via Apache Ivy

[simon04]

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

[Don-vip]

Replying to simon04:

In 16025/josm:

see #16860 - Resolve com.drewnoakes:metadata-extractor via Apache Ivy

Are you sure it works for this one? I remember I must patch it at each update.

[simon04]

Here's how the dist sizes evolved:

ls -l --time-style full-iso 
total 84764
-rw-r--r-- 1 simon users 14443614 2020-03-03 22:21:13 +0100 josm-custom-16016.jar
-rw-r--r-- 1 simon users 14444368 2020-03-03 22:23:10 +0100 josm-custom-16016+json.jar
-rw-r--r-- 1 simon users 14429230 2020-03-03 22:24:45 +0100 josm-custom-16016+json+xz.jar
-rw-r--r-- 1 simon users 14432267 2020-03-03 22:37:35 +0100 josm-custom-16016+json+xz+compress.jar
-rw-r--r-- 1 simon users 14430688 2020-03-03 22:40:41 +0100 josm-custom-16016+json+xz+compress+logging.jar
-rw-r--r-- 1 simon users 14604031 2020-03-03 23:28:36 +0100 josm-custom-16016+json+xz+compress+logging+metadata-extractor.jar

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.

---

How to proceed?

• JMapViewer?
• commons-jcs: our SVN version includes 54 commits since the latest release, see ​https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master
• signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746
• svgSalamander: keep it for now

[simon04]

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

[Don-vip]

Replying to simon04:

The increase for metadata-extractor is due to our disabled imports in JpegMetadataReader.java which cannot be ported to Apache Ivy.

We could ask the author to provide smaller artifacts.

[Don-vip]

Replying to simon04:

• signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

Did we consider ​https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

[Don-vip]

Replying to simon04:

Replying to Don-vip:

Are you sure it works for this one? I remember I must patch it at each update.

Haven't found any problem yet. I successfully opened a bunch of JPG images; ExifReaderTest passes.

I'm almost sure it doesn't work. Look at proguard output:

 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.quicktime.QuickTimeTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.riff.RiffTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.FileTypeDetector: can't find referenced class com.drew.imaging.mp3.MpegAudioTypeChecker
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.psd.PsdMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.png.PngMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.bmp.BmpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.gif.GifMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.ico.IcoMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.pcx.PcxMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.webp.WebpMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.raf.RafMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.avi.AviMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.wav.WavMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.quicktime.QuickTimeMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp4.Mp4MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.mp3.Mp3MetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.eps.EpsMetadataReader
 [proguard] Warning: com.drew.imaging.ImageMetadataReader: can't find referenced class com.drew.imaging.heif.HeifMetadataReader
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$1: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamProvider
 [proguard] Warning: org.apache.commons.compress.compressors.CompressorStreamFactory$2: can't find referenced class org.apache.commons.compress.compressors.CompressorStreamFactory
 [proguard] Warning: org.apache.commons.compress.compressors.xz.XZUtils$CachedAvailability: can't find referenced class org.apache.commons.compress.compressors.xz.XZUtils
 [proguard] Warning: there were 61 unresolved references to classes or interfaces.

[simon04]

Replying to Don-vip:

Did we consider ​https://github.com/scribejava/scribejava at some point? It looks way more active, but probably heavier.

Half an hour ago, I've also found this library. Looks interesting. Will investigate in the next days. The first thing I noticed is that com.github.scribejava.core.extractors.AbstractOAuth1JSONTokenExtractor makes use of Jackson for JSON decoding, but we could probably provide our own implementation.

[simon04]

Replying to Don-vip:

I'm almost sure it doesn't work. Look at proguard output:

This is the list of classes which are now included in josm.jar, but haven't been before:

$ comm -3 all-classes-in-josm.jar-before all-classes-in-josm.jar-after
        com/adobe/
        com/adobe/internal/
        com/adobe/internal/xmp/
        com/adobe/internal/xmp/impl/
        com/adobe/internal/xmp/impl/Base64.class
        com/adobe/internal/xmp/impl/ByteBuffer.class
        com/adobe/internal/xmp/impl/CountOutputStream.class
        com/adobe/internal/xmp/impl/FixASCIIControlsReader.class
        com/adobe/internal/xmp/impl/ISO8601Converter.class
        com/adobe/internal/xmp/impl/Latin1Converter.class
        com/adobe/internal/xmp/impl/ParameterAsserts.class
        com/adobe/internal/xmp/impl/ParseRDF.class
        com/adobe/internal/xmp/impl/ParseState.class
        com/adobe/internal/xmp/impl/QName.class
        com/adobe/internal/xmp/impl/Utils.class
        com/adobe/internal/xmp/impl/XMPDateTimeImpl.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator$1.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIteratorChildren.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl$NodeIterator.class
        com/adobe/internal/xmp/impl/XMPIteratorImpl.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$1.class
        com/adobe/internal/xmp/impl/XMPMetaImpl$2.class
        com/adobe/internal/xmp/impl/XMPMetaImpl.class
        com/adobe/internal/xmp/impl/XMPMetaParser.class
        com/adobe/internal/xmp/impl/XMPNode$1.class
        com/adobe/internal/xmp/impl/XMPNode.class
        com/adobe/internal/xmp/impl/XMPNodeUtils.class
        com/adobe/internal/xmp/impl/XMPNormalizer.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl$1.class
        com/adobe/internal/xmp/impl/XMPSchemaRegistryImpl.class
        com/adobe/internal/xmp/impl/XMPSerializerHelper.class
        com/adobe/internal/xmp/impl/XMPSerializerRDF.class
        com/adobe/internal/xmp/impl/XMPUtilsImpl.class
        com/adobe/internal/xmp/impl/xpath/
        com/adobe/internal/xmp/impl/xpath/PathPosition.class
        com/adobe/internal/xmp/impl/xpath/XMPPath.class
        com/adobe/internal/xmp/impl/xpath/XMPPathParser.class
        com/adobe/internal/xmp/impl/xpath/XMPPathSegment.class
        com/adobe/internal/xmp/options/
        com/adobe/internal/xmp/options/AliasOptions.class
        com/adobe/internal/xmp/options/IteratorOptions.class
        com/adobe/internal/xmp/options/Options.class
        com/adobe/internal/xmp/options/ParseOptions.class
        com/adobe/internal/xmp/options/PropertyOptions.class
        com/adobe/internal/xmp/options/SerializeOptions.class
        com/adobe/internal/xmp/properties/
        com/adobe/internal/xmp/properties/XMPAliasInfo.class
        com/adobe/internal/xmp/properties/XMPProperty.class
        com/adobe/internal/xmp/properties/XMPPropertyInfo.class
        com/adobe/internal/xmp/utils/
        com/adobe/internal/xmp/utils/XMLStreamWriterFactory.class
        com/adobe/internal/xmp/utils/XMLStreamWriterImpl.class
        com/adobe/internal/xmp/XMPConst.class
        com/adobe/internal/xmp/XMPDateTime.class
        com/adobe/internal/xmp/XMPDateTimeFactory.class
        com/adobe/internal/xmp/XMPError.class
        com/adobe/internal/xmp/XMPException.class
        com/adobe/internal/xmp/XMPIterator.class
        com/adobe/internal/xmp/XMPMeta.class
        com/adobe/internal/xmp/XMPMetaFactory$1.class
        com/adobe/internal/xmp/XMPMetaFactory.class
        com/adobe/internal/xmp/XMPPathFactory.class
        com/adobe/internal/xmp/XMPSchemaRegistry.class
        com/adobe/internal/xmp/XMPUtils.class
        com/adobe/internal/xmp/XMPVersionInfo.class
        com/drew/imaging/FileType.class
        com/drew/imaging/FileTypeDetector.class
        com/drew/imaging/ImageMetadataReader$1.class
        com/drew/imaging/ImageMetadataReader.class
        com/drew/imaging/tiff/TiffMetadataReader.class
        com/drew/imaging/TypeChecker.class
        com/drew/lang/ByteConvert.class
        com/drew/lang/ByteTrie$ByteTrieNode.class
        com/drew/lang/ByteTrie.class
        com/drew/lang/ByteUtil.class
        com/drew/lang/Iterables.class
        com/drew/lang/KeyValuePair.class
        com/drew/lang/NullOutputStream.class
        com/drew/lang/RandomAccessFileReader.class
        com/drew/lang/RandomAccessStreamReader.class
        com/drew/lang/StreamUtil.class
        com/drew/metadata/adobe/
        com/drew/metadata/adobe/AdobeJpegDescriptor.class
        com/drew/metadata/adobe/AdobeJpegDirectory.class
        com/drew/metadata/adobe/AdobeJpegReader.class
        com/drew/metadata/file/FileTypeDescriptor.class
        com/drew/metadata/file/FileTypeDirectory.class
        com/drew/metadata/jfif/
        com/drew/metadata/jfif/JfifDescriptor.class
        com/drew/metadata/jfif/JfifDirectory.class
        com/drew/metadata/jfif/JfifReader.class
        com/drew/metadata/jfxx/
        com/drew/metadata/jfxx/JfxxDescriptor.class
        com/drew/metadata/jfxx/JfxxDirectory.class
        com/drew/metadata/jfxx/JfxxReader.class
        com/drew/metadata/Schema.class
        com/drew/metadata/xmp/
        com/drew/metadata/xmp/XmpDescriptor.class
        com/drew/metadata/xmp/XmpDirectory.class
        com/drew/metadata/xmp/XmpReader.class
        com/drew/metadata/xmp/XmpWriter.class

In particular, FileTypeDetector and ImageMetadataReader are present in this list. We could exclude all those …

[simon04]

In 16026/josm:

see #16860 - Apache Ivy: exclude various subclasses from inclusion

[simon04]

In 16027/josm:

see #16860 - Apache Ivy: exclude com/drew/imaging/{FileTypeDetector,ImageMetadataReader} from inclusion

[Don-vip]

In 16050/josm:

see #18845 - see #16860 - directory cleanup

[simon04]

Replying to Don-vip:

Replying to simon04:

• signpost: we did some minor cleanups and kicked out commons-codecs in r8149, r10746

Signpost is dead, no commit in 5 years, 15 open pull requests...

I posted to the signpost mailing list: ​https://groups.google.com/forum/#!msg/signpost-users/QDccoNNTvpc/5pRpdLZyAAAJ

[simon04]

Hi sebastic, this might affect the packaging for Debian. A few libraries are obtained as JAR via Apache Ivy, see source:trunk/ivy.xml​

[sebastic]

move JMapViewer outside OSM SVN server

Where is it going? There is no JMapViewer repo on GitHub yet.

The Debian package uses the ZIP releases from ​https://svn.openstreetmap.org/applications/viewer/jmapviewer/releases/.

In the ivy config I see metadata-extractor, we used to build the josm package with the separately packaged metadata-extractor but because its API is highly unstable updating always broke either josm or one of the other dependent packages.

If the sources for unstable dependencies like this won't be bundled with josm any more, we won't be able to keep to the josm package in Debian.

[simon04]

The goal is to use the official JAR distributions for external dependencie, and not (partially) compile them within the JOSM build system.

The current status:

• Apache Commons Compress ✔
• Apache Commons JCS: TODO, needs new release, see comment:24
• Apache Commons Logging ✔
• SvgSalamander: UNLIKELY
• Metadata Extractor ✔
• Signpost: TODO, WIP, see comment:34
• Jsonp ✔
• OpeningHoursValidator ✔
• JMapViewer: TODO (needs separate repository first)

Running ant dist depends on ant resolve (which runs Apache Ivy to fetch the dependencies) and bundles the dependencies in a fatjar. So, by default all dependencies are bundled into dist/josm-custom.jar.

[sebastic]

ant resolve doesn't work in a Debian build environment as it doesn't have network, the dependencies need to have their source included in the source tree or an appropriate version needs to be packaged and installed in the build environment.

Experience has shown that using separately packaged metadata-extractor and svgsalamander made backporting JOSM tested for Debian stable a PITA. Once the embedded copies were used this was much less of a problem. The separately packaged JMapViewer has also broken freeplane on more than one occasion, forcing stable users to choose whether to have the josm backport installed (which requires a newer jmapviewer) or a working freeplane.

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

[simon04]

Is there a Debian compatible way not involving svn:externals? By no means, the goal was to make Debian packging harder/impossible. However, slowly getting rid of svn:externals allows us to move forward in the long-term goal #16871.

[sebastic]

Publish a source jar that bundles the required versions of the dependencies?

[simon04]

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

[wiktorn]

Replying to sebastic:

Because these recent changes make building JOSM from source in a Debian build environment unfeasible when backports are required as well, we'll have to remove the josm package from Debian and replace it with an installer that downloads the pre-built JAR. This is not very good either (because the JARs don't have hashes nor cryptographic signatures to verify the download with), but at least it will keep JOSM tested easily available to users of Debian stable (assuming the josm-installer package passes FTP master review).

Jars published on JOSM page are signed and it is possibile to verify the signature. See: ​https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

If I remember correctly, it was possibile to download sources in Ivy

[Don-vip]

Another Ivy issue we need to fix: proxy support, see ​https://lists.openstreetmap.org/pipermail/josm-dev/2020-March/008288.html

[sebastic]

Replying to wiktorn:

Jars published on JOSM page are signed and it is possibile to verify the signature. See: ​https://docs.oracle.com/en/java/javase/11/tools/jarsigner.html

That's not a equivalent to ​detached GPG signatures.

jarsigner is part of the JDK, installing the entire JDK to run JOSM is highly undesirable.

[Don-vip]

In 16118/josm:

see #16860 - instruct Eclipse IvyDE plugin to use ivysettings.xml

[Don-vip]

In 16125/josm:

see #16860 - see #18880 - explicit use of Ivy configuration files for Netbeans compatibility

[Don-vip]

In 16139/josm:

see #16860 - see #18880 - setup Ivy in Netbeans project

[simon04]

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

[simon04]

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - ​https://issues.apache.org/jira/browse/JCS-204

[Don-vip]

Replying to simon04:

In 16141/josm:

see #16860 - build.xml: add ant sources

Generates jar file of JOSM source files and its dependencies.

@sebastic: I guess you don't need the REVISION file?

@simon: Dropping the "create-revision" dependency would make it easier for me to include this new task in the server Makefile

[sebastic]

Replying to Don-vip:

@sebastic: I guess you don't need the REVISION file?

The package already creates it itself:

​https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/rules#L80

And it's patched to include Build-Name & Debian-Release:

​https://salsa.debian.org/debian-gis-team/josm/-/blob/debian/0.0.svn15937+dfsg-1/debian/patches/00-build.patch#L39

[simon04]

In 16150/josm:

see #16860 - build.xml: ant sources: remove REVISION

[Don-vip]

@sebastic can you please check if https://josm.openstreetmap.de/download/josm-latest-sources.jar is enough for you to build a JOSM Debian package.

[sebastic]

JOSM tested 15937 contained these subdirectories under src/ which are not included in the source JAR:

• javax/json
• org/jdesktop

[simon04]

In 16157/josm:

see #16860 - build.xml: ant sources: use javax.json:javax.json-api

[simon04]

• org/jdesktop is obsolete due to r16050

[simon04]

In 16158/josm:

see #16860 - Apache Ivy: re-add org.glassfish:javax.json

[simon04]

In 16165/josm:

see #16860 - Resolve PMD using Apache Ivy

[Don-vip]

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

[Don-vip]

Also: don't forget plugins. build-common.xml has references to these jar files.

[Don-vip]

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

[simon04]

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one. All others can be retrieved from Ivy I think.

Alright :)

[simon04]

In 16166/josm:

see #16860 - Resolve SpotBugs using Apache Ivy

[simon04]

In 16167/josm:

see #16860 - Separate tools/ivy.xml

[simon04]

[o35377] - see #josm16860 - Resolve SpotBugs using Apache Ivy

[simon04]

In 16168/josm:

see #16860 - Resolve JavaCC using Apache Ivy

[simon04]

[o35378] - see #josm16860 - Resolve JavaCC using Apache Ivy

[simon04]

In 16169/josm:

see #16860 - Resolve ProGuard using Apache Ivy

[simon04]

In 16171/josm:

see #16860 - Resolve Checkstyle using Apache Ivy

[simon04]

[o35379] - see #see16860 - Resolve Checkstyle using Apache Ivy

[sebastic]

Replying to Don-vip:

@sebastic: I guess you don't need any of these test/static analysis tools for Debian packaging?

Only if they're called from build.xml via the dist target dependency chain.

[skyper]

Not sure if this is the ticket to blame but there was no latest built this morning.

[Don-vip]

Replying to skyper:

Not sure if this is the ticket to blame but there was no latest built this morning.

Yep, the server Makefile didn't like r16168. Looking into it

[Don-vip]

Fixed. But r16165 broke ImageryCompare, looking into this now.

EDIT: fixed

[Don-vip]

In 16176/josm:

see #16860 - update Eclipse classpath

[Don-vip]

@Simon: ant spotbugs is broken for plugins:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:60: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:666: impossible to build ivy path: bad confs provided: spotbugs not found among [default]

[simon04]

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

[simon04]

I would like to get rid of source:trunk/tools/ivy/ivy.jar in the repository.

Variant 1: require Apache Ivy to be installed systemwide (as we do for Apache Ant)

Variant 2: download from https://josm.openstreetmap.de/nexus/content/repositories/public/ (as we do for plugins, as is shown in the ​most basic Ivy example)

build.xml

@@ -16 +16 @@
          xmlns:unless="ant:unless"
 >
     <target name="init-ivy">
+        <property name="ivy.version" value="2.5.0"/>
         <dirname property="base.dir" file="${ant.file.josm}"/>
         <property name="lib.dir"   location="${base.dir}/lib"/>
         <property name="tools.dir" location="${base.dir}/tools"/>
         <property name="tools.ivy" location="${tools.dir}/ivy.xml"/>
-        <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${tools.dir}/ivy/ivy.jar"/>
+        <property name="ivy.jar.dir" location="${tools.dir}/ivy"/>
+        <property name="ivy.jar.file" location="${ivy.jar.dir}/ivy-${ivy.version}.jar"/>
+        <mkdir dir="${ivy.jar.dir}"/>
+        <get src="https://josm.openstreetmap.de/nexus/content/repositories/public/org/apache/ivy/ivy/${ivy.version}/ivy-${ivy.version}.jar"
+             dest="${ivy.jar.file}"
+             skipexisting="true"
+        />
+        <echo>Apache Ivy ${ivy.jar.file}</echo>
+        <taskdef resource="org/apache/ivy/ant/antlib.xml" uri="antlib:org.apache.ivy.ant" classpath="${ivy.jar.file}"/>
     </target>
     <target name="init-properties" depends="resolve">
         <property environment="env"/>

Which variant should we take?

[Don-vip]

I would say 2. Less disruptive for users. Probably as a new step that can be skipped for Debian build, or any system that provides ivy?
@sebastic your opinion?

[sebastic]

No preference, the Debian package build won't call the ivy target.

[Don-vip]

Replying to simon04:

I cannot reproduce locally. So I'm groping in the dark. Maybe [o35389] helps.

[o35389] - see ​#josm16860 - Add ant resolve-tools target

[o35390] - see ​#josm16860 - Update to Apache Ivy 2.5.0

It's... different :)

resolve-tools:

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:57: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:727: java.lang.NullPointerException
	at org.apache.ivy.ant.IvyAntSettings.getDefaultProperties(IvyAntSettings.java:322)
	at org.apache.ivy.ant.IvyAntSettings$1.execute(IvyAntSettings.java:268)
	at org.apache.ivy.ant.IvyAntSettings.createIvyEngine(IvyAntSettings.java:273)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:139)
	at org.apache.ivy.ant.IvyAntSettings.getDefaultInstance(IvyAntSettings.java:150)
	at org.apache.ivy.ant.IvyTask.getIvyInstance(IvyTask.java:77)
	at org.apache.ivy.ant.IvyTask.prepareTask(IvyTask.java:237)
	at org.apache.ivy.ant.IvyResolve.prepareTask(IvyResolve.java:236)
	at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:258)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
	at org.apache.tools.ant.Task.perform(Task.java:350)
	at org.apache.tools.ant.Target.execute(Target.java:449)

[Don-vip]

The JDK11 build passed, I'm relaunching the JDK8 build.

[simon04]

In 16183/josm:

see #16860 - <get> Apache Ivy, remove ivy.jar from repository

[simon04]

Replying to simon04:

I've asked the apache-jcs maintainers to craft a new release which we could use via Apache Ivy: [JCS-204] Craft a new 2.3 release - ​https://issues.apache.org/jira/browse/JCS-204

Thomas Vandahl commented on JCS-204:

Unfortunately, the current trunk/master contains lots of breaking changes, so that a new release *must* be 3.x. This will require the package name as well as the Maven coordinates to reflect this. I'll see what I can do.

Update: I renamed the ticket to [JCS-204] Release JCS 3.0 - ASF JIRA

[simon04]

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

[Don-vip]

Replying to simon04:

@Vincent, how should we proceed with commons-jcs and jmapviewer?

Ad commons-jcs: could we publish the current SNAPSHOT to JOSM's Nexus and use 3.0-SNAPSHOT from there?

Is 3.0-SNAPSHOT already available?

Ad jmapviewer: the latest release in JOSM's Nexus is 2.9, the latest released version 2.13 – Can we publish 2.10…2.13 to Nexus and include 2.13 via Apache Ivy

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

[Don-vip]

Replying to Don-vip:

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

I have changed the Jenkins job to take VERSION as a parameter, seems to work. We can keep it is it until we switch to Gitlab I think. All versions should be available in Nexus now.

[simon04]

Replying to Don-vip:

Is 3.0-SNAPSHOT already available?

3.0-SNAPSHOT is the version we are currently using via svn:externals, i.e., ​https://github.com/apache/commons-jcs/commit/366e4f001aaa72cf489f25d642ff88648a81a3fe
Can we compile and publish this version to our Nexus?

The jmapviewer nexus deploy currently relies on a very unreliable Jenkins job: https://josm.openstreetmap.de/jenkins/job/Nexus-JMapViewer/
We should make something most robust. Ideally from Ant? Something like an "ant release" target? For the old releases I can import them manually.

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

[Don-vip]

Replying to simon04:

Can we compile and publish this version to our Nexus?

I setup an automatic build&deploy job: https://josm.openstreetmap.de/jenkins/job/Nexus-JCS/7/console
Can you please test?

Thank you. Shall we migrate to jmapviewer 2.13 via Apache Ivy now?

Sure!

Having accomplished those two steps would allow us to eliminate the svn:externals completely. :))

\o/

[simon04]

In 16194/josm:

see #16860 - Resolve JMapViewer using Apache Ivy

[simon04]

In 16195/josm:

see #16860 - Resolve commons-jcs-core using Apache Ivy

[simon04]

Awesome, thank you! Seems to be working like a charm (at least on my system). In r16195 I had to remove the <exclude>s (i.e., not migrate them to extract-libraries) as otherwise spurious JCS/Cache exceptions would show up.

[Don-vip]

Proguard fails.

[simon04]

In 16197/josm:

see #16860 - Apache Ivy: exclude various commons-jcs classes from inclusion

[simon04]

Replying to Don-vip:

Proguard fails.

The usual suspect ;)

[skyper]

Is this ticket to blame that there was no latest released this morning ?

[anonymous]

Replying to skyper:

Is this ticket to blame that there was no latest released this morning ?

no latest today, should be 16199.

[Don-vip]

Ticket not to blame, random network error. Tomorrow it should work.

[simon04]

In 16209/josm:

see #16860 - directory cleanup

[GerdP]

After updating to r16211 I can no longer compile in Eclipse. I tried

ant clean dist

and a refresh in Eclipse but that did not help. I've also restarted Eclipse. I still have 1309 compile errors.

[GerdP]

Seems all sources containing import org.openstreetmap.gui.* fail.

[GerdP]

Fixed, had to do ivy resolve manually in Eclipse

[Bjoeni]

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

[Klumbumbus]

In 16219/josm:

see #16860, see #18880 - Adapt Netbeans config to r16165

[Klumbumbus]

After r16166 Netbeans can't find spotbugs.jar anymore and the build fails, probably also due to r16194, as the message is:

C:\Users\stefa\Documents\OSM\josm\core\ide\netbeans\nbbuild.xml:90: C:\Users\stefa\Documents\OSM\josm\core\src\org\openstreetmap\gui\jmapviewer\images does not exist.
BUILD FAILED (total time: 1 minute 34 seconds)

(see also #18880)

[taylor.smock]

Replying to Bjoeni:

Disclaimer: I didn't really follow / read all of this ticket and I never worked with Apache Ivy before (I only stumbled across this ticket because org.openstreetmap.gui.* failed to resolve).

But what's the best way to use JMapViewer in a JOSM plugin at the moment? I installed IvyDE in Eclipse and had it resolve, it added a reference to jmapviewer-2.13.jar. I then added that jar (from the Ivy cache directory) manually to the referenced libraries of my plugin. Is that the way to go as of right now or am I just not getting how Ivy is supposed to work?

I've had much the same issue. This (also) probably isn't the best way to go about it, but I've been changing the build path. So "Configure build path" -> "Order and Export" -> Check "Ivy". I've done this since most of my projects use JSON, and javax.json is now resolved by Ivy.

If you do that, you will have a diff for .classpath that should look like this:

.classpath

@@ -32 +32 @@
                         <attribute name="test" value="true"/>
                 </attributes>
         </classpathentry>
-        <classpathentry kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
+        <classpathentry exported="true" kind="con" path="org.apache.ivyde.eclipse.cpcontainer.IVYDE_CONTAINER/?project=JOSM&amp;ivyXmlPath=ivy.xml&amp;confs=*&amp;ivySettingsPath=ivysettings.xml&amp;loadSettingsOnDemand=false&amp;ivyUserDir=&amp;propertyFiles="/>
         <classpathentry kind="lib" path="test/lib/jfcunit.jar">
                 <attributes>
                         <attribute name="test" value="true"/>

[Don-vip]

In 16221/josm:

see #16860 - remove copy of jmapviewer images in Netbeans project

[Don-vip]

@Klumbumbus can you please test Netbeans?

[Klumbumbus]

The build works again, but it still can't find spotbugs.jar.

[Don-vip]

In 16222/josm:

see #16860 - update spotbugs jar in Netbeans project

[Don-vip]

Done. Please check.

[Klumbumbus]

👍

[Don-vip]

Something happened between r16155 and r16200 that broke the build for Java 14/15, but not for Java 8/11.
This is the output we get with Java 14+:

test-compile:
    [javac] Compiling 569 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/unit
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning
    [javac] Compiling 19 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/functional
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning
    [javac] Compiling 12 source files to /var/lib/jenkins/jobs/Java-EarlyAccess-JOSM/workspace/jdk/JDK14/test/build/performance
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 8
    [javac] 1 warning

test:
     [echo] Running unit tests with JUnit
[jacoco:coverage] Enhancing junit with coverage
    [junit] Error: A JNI error has occurred, please check your installation and try again
    [junit] Running org.openstreetmap.josm.tools.template_engine.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)
     [echo] Running functional tests with JUnit
[jacoco:coverage] Enhancing junit with coverage
    [junit] Error: A JNI error has occurred, please check your installation and try again
    [junit] Running org.openstreetmap.josm.tools.Batch-With-Multiple-Tests
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests FAILED (crashed)

[simon04]

I cannot reproduce locally with r16225

$ gradle --version
...
Ant:          Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM:          14 (AdoptOpenJDK 14+36)

$ ant clean

$ ant test '-Ddefault-junit-includes=**/josm/tools/**/*Test.class'
...
test:
     [echo] Running unit tests with JUnit
...
BUILD SUCCESSFUL
Total time: 1 minute 11 seconds

[simon04]

Replying to Don-vip:

@Simon: if you begin to resolve tools via Ivy, please keep error_prone, no official version actually works, a lot of patches are needed. I'm also using a Jacoco snapshot (unpatched) for Java 15 compatibility, so please wait the next release for this one.

Can we deploy those snapshots to https://josm.openstreetmap.de/nexus/content/repositories/snapshots/ as we've done for Apache Commons JCS? Then we could also resolve all test dependencies using Apache Ivy.

JaCoCo got official Java 14 support in the most recent commit (with the nice commit message "Happy birthday JDK 14!"): ​https://github.com/jacoco/jacoco/commit/1c5601bba5bf195a5f9b09a87cb5d2152857bbb7#diff-0886964cfc7b181768d1d9b182f50528R25

[Don-vip]

Ah you're right it's just that I enabled Jacoco coverage in r16156. Strangely it works also for me locally, even in headless mode.

[Don-vip]

Replying to simon04:

Can we deploy those snapshots to https://josm.openstreetmap.de/nexus/content/repositories/snapshots/ as we've done for Apache Commons JCS?

No need, they have their own maven repo for snapshots.

Then we could also resolve all test dependencies using Apache Ivy.

I'm trying to do it :)

[Don-vip]

Replying to Don-vip:

I'm trying to do it :)

I have bad luck: the very moment I try to use their snapshots repository, it goes down: ​https://status.maven.org/incidents/bwklmrfj9z1c

[simon04]

Replying to Don-vip:

No need, they have their own maven repo for snapshots.

Convenient :-)

Replying to Don-vip:

I have bad luck: the very moment I try to use their snapshots repository, it goes down

:-D
It's up again:

Quoting ​https://status.maven.org/incidents/bwklmrfj9z1c

Services have been restored on oss.sonatype.org.
Posted 11 minutes ago. Apr 04, 2020 - 09:20 EDT

[Don-vip]

In 16229/josm:

see #16860 - resolve test libraries using Ivy

[Don-vip]

In 16231/josm:

see #16860 - fix compilation of scripts

[Don-vip]

Found out what the Java 14 issue was. It came from the Jenkins job configuration:

jacoco.includes=org.openstreetmap.josm.*:java.*:javax.*:com.sun.*:sun.*
jacoco.inclbootstrapclasses=true
jacoco.inclnolocationclasses=true

I think I tried a long time ago to cover JDK classes too, when we were part of the OpenJDK Outreach, before Oracle decided to go full $$$ on Java.
I removed this configuration and now it works :)

[Don-vip]

Plugin build updated in [o35411:35412].

[simon04]

[o35413] - resolve test libraries using Ivy settings from core

[GerdP]

Is there an easy way to suppress all the [ivy:resolve] messages?
When I use e.g.

ant checkstyle pmd spotbugs

it is difficult to find the messages from checkstyle and checkstyle-josm.xml is hard to read because of its length.
I also wonder why I see those for each target. Is that intended?
For now I try to get used to a different order:

ant pmd spotbugs checkstyle 

[simon04]

In 16243/josm:

see #16860 - Apache Ivy: tune logging

[GerdP]

thanks!

[sebastic]

Replying to simon04:

The josm-sources.jar would contain all JOSM's own sources (.java files + resources) and all of its dependencies (plenty of additional .java files + resources)? This seems doable, but requires some investigations into Ant/Ivy (at least, I don't know how to accomplish this by heart).

The current source JAR is not sufficient, there are some transitive dependencies, to build the subset of metadata-extractor required for JOSM the ​adobe-xmp-core sources are also required, but it's ​license terms are non-free i.e. incompatible with the DFSG & OSD.

The OpeningHoursParser sources also cannot be built because the ParseException & Token symbols cannot be found. This seems to be missing imports. It may also be related to using JDK 11 to build JOSM whereas OpeningHoursParser likely still uses JDK 8.

Update: Compiling the metadata-extractor sources was possible after adding the sources from ​xmpcore-6.0.6-sources.jar (BSD-3-Clause), and compiling the OpeningHoursParser sources after adding the sources from ​annotations-19.0.0-sources.jar (Apache-2.0) and a few generated sources.

[simon04]

In 16338/josm:

see #16860 - Apache Ivy: update xmpcore to fix warning

[simon04]

In 16339/josm:

see #16860 - Apache Ivy: remove unused epsg configuration

[GerdP]

Please check https://josm.openstreetmap.de/jenkins/job/JOSM-Plugins/jdk=JDK8/1842/console

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:54: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build.xml:29: The following error occurred while executing this line:
/var/lib/jenkins/jobs/JOSM-Plugins/workspace/jdk/JDK8/build-common.xml:551: impossible to ivy retrieve: java.lang.RuntimeException: problem during retrieve of org.openstreetmap.josm.plugins#wikipedia: java.lang.IllegalStateException: Report file '/var/lib/jenkins/.ant/cache/org.openstreetmap.josm.plugins-wikipedia-test.xml' does not exist.

[stoecker]

Build in plugins fails:

~/josm/plugins/apache-commons> ant
Buildfile: /home/stoecker/sources/svn.openstreetmap.org/applications/editors/josm/plugins/apache-commons/build.xml

resolve:

BUILD FAILED
/home/stoecker/sources/svn.openstreetmap.org/applications/editors/josm/plugins/apache-commons/build.xml:23: Problem: failed to create task or type antlib:org.apache.ivy.ant:retrieve
Cause: The name is undefined.
Action: Check the spelling.
Action: Check that any custom tasks/types have been declared.
Action: Check that any <presetdef>/<macrodef> declarations have taken place.
No types or tasks have been defined in this namespace yet

This appears to be an antlib declaration. 
Action: Check that the implementing library exists in one of:
        -/usr/share/ant/lib
        -/home/stoecker/.ant/lib
        -a directory added on the command line with the -lib argument


Total time: 0 seconds

[simon04]

In 16403/josm:

see #16860 - Resolve oauth.signpost:signpost-core via Apache Ivy

Signpost 2.0.0 includes the update to Java 8: ​https://github.com/mttkay/signpost/releases/tag/oauth-signpost-2.0.0

[GerdP]

plugin pdfimport is missing org.apache.commons.logging and others:
https://josm.openstreetmap.de/jenkins/job/JOSM-Plugins/jdk=JDK8/1850/consoleText
I assume something reg. ivy is missing?

[GerdP]

I can compile the plugin when I uncomment the line in the build.xml

<!-- <property name="plugin.requires" value="apache-commons"/> -->

Why is this line needed again now? It was commented in 2018 and the build worked in April 2020

[taylor.smock]

Possibly. It looks like there used to be a src/org/apache/commons/logging directory in the source code, which is why the plugin was compiling correctly (the logging library removed in r16050).

Do you know if there was a reason for using a logging library directly instead of JOSM Logging?

[GerdP]

I don't understand what's going on. I've reverted my changes and now ant clean dist works in directory pdfimport.

[GerdP]

.. and now its back again. Seems a dependency is missing?

[simon04]

commons-logging via Ivy: r16024
commons-logging dropped as it is nowhere used in JOSM core: r16397

[GerdP]

So, what should be done with plugins? Is

<property name="plugin.requires" value="apache-commons"/> 

the solution?

[taylor.smock]

It is probably the quickest solution.

But all the pdfimport plugin is doing is creating a log instance (with Apache Logging LogFactory), and then logging exceptions to it (log.warn). I think that pdfimport can, at least, get by without using the Apache Logging library. I think that the loggers could be replaced with Logging.warn(e) with no loss of functionality.

[simon04]

pdfimport: removed commong.logging usages in [o35448:35449].

However, pdfbox internally uses log4j according to ​https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/1.8.12 and the following warning is printed (I can live with that):

log4j:WARN No appenders could be found for logger (org.apache.pdfbox.pdfparser.PDFObjectStreamParser).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

[simon04]

In 16408/josm:

see #16860 - build.xml: remove leftovers of org.apache.commons.logging

[Don-vip]

In 16536/josm:

see #16860 - remove leftovers of OAuth signpost sources

[Don-vip]

Can't we remove com/google/gdata/util/common/base? According to r4231 it seems used by signpost and/or metadata extractor, both are managed by Ivy now.

[simon04]

In 16537/josm:

see #16860 - Remove src/com/google/gdata

Leftover from r16403

[Don-vip]

Only one dependency to go \o/

[simon04]

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

[Don-vip]

Replying to simon04:

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

If you feel it's stable enough, let's do it. I didn't follow JOSM development on last month.

[stoecker]

Replying to Don-vip:

Replying to simon04:

The hardest one … Might not be ready for the 20.05 milestone … ;-)

Should we release r16538 as 20.05?

If you feel it's stable enough, let's do it. I didn't follow JOSM development on last month.

You have to wait till tomorrow, as this version lies behind the nightly build. I've not seen anything major in the new tickets lately, so I also have no objections. I actually wanted to set sundays version but there were new checkins all the time.

[simon04]

• r16536 is the built latest version.
• r16537 is irrelevant for the release.
• r16538 is an i18n update of a few languages.

If we can trigger a latest build for r16538, let's take this one. Otherwise r16536 is good. :-)

Thanks!

[stoecker]

We waited so long, we can wait another day :-)

[Don-vip]

I just launched a manual build :)

[simon04]

In 16603/josm:

see #16860 - Suppress "bad path element .../xmpcore-6.0.6.jar: no such file or directory" warning

[simon04]

In 17128/josm:

see #16860 - Apache Ivy: only use josm-nexus repository by default

[GerdP]

What are the required steps to configure ivy so that it doesn't print lots of these error messages when compiling a plugin like e.g.. reverter?
[ivy:resolve] unknown resolver josm-nexus

[GerdP]

In the wikipedia plugin directory I find this ivy_settings.xml :

<?xml version="1.0" encoding="utf-8"?>
<ivysettings>
  <settings defaultResolver="josm-nexus"/>
  <resolvers>
    <ibiblio name="josm-nexus" m2compatible="true" root="https://josm.openstreetmap.de/nexus/content/repositories/public/" />
  </resolvers>
</ivysettings>

When I copy this file into the reverter directory the messages disappear. Does that mean we need this file in every plugin directory that doesn't yet have one? Or is it possible to have a working default for all of them?

[Don-vip]

In 35657/osm:

see #16860 - upgrade to JNA 5.6.0 + switch to Ivy

[simon04]

In 35764/osm:

see #16860 - ivy_settings: consistently use "josm-nexus"

Try to solve "unknown resolver josm-nexus" warnings in Jenkins.

[simon04]

In 35766/osm:

see #16860 - Apache Ivy: fallback to core ivysettings.xml when plugin ivy_settings.xml does not exist

[simon04]

An attempt to merge ivy.xml and tools/ivy.xml – ​https://github.com/openstreetmap/josm/compare/one-ivy

Rationale: one file for all dependencies. javacc and errorprone are currently listed in tools/ivy.xml, but are needed for compiling JOSM.