commons-io is required for plugins but not for JOSM

[sebastic]

As reported in ​Debian Bug #1087811 the MapWithAI plugin fails because commons-io classes are not found on the classpath:

java.lang.ClassNotFoundException: org.apache.commons.io.input.BoundedInputStream 

I've had similar issues with the wikipedia plugin which I've disabled since.

Since JOSM has no direct dependency on commons-io, should the apache-commons plugin be updated to include these classes?

[taylor.smock]

Looking at the bug report, I think the important lines are as follows:

By running JOSM with -verbose:class, I noticed that
GzipCompressorInputStream gets loaded from the system-wide version and
not from the plugin:

[28,500s][info][class,load] 
org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream 
source: file:/usr/share/java/commons-compress.jar

In contrast, when running JOSM’s official release, it gets loaded from
the plugin

[25,273s][info][class,load] 
org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream 
source: file:/home/user/.local/share/JOSM/plugins/apache-commons.jar

I also found it noteworthy that BoundedInputStream is contained in the
official JOSM JAR, but not Debian’s JAR?

The missing class is in the apache-commons.jar file (as noted in the bug report). When I built a version of JOSM without BoundedInputStream in the jar file, the class was properly loaded from apache-commons.jar

[19.111s][info][class,load] org.apache.commons.io.input.BoundedInputStream source: file:/Users/tsmock/Library/JOSM/plugins/apache-commons.jar

pom.xml

@@ -608 +608 @@
             <configuration>
               <minimizeJar>true</minimizeJar>
               <filters>
+                <filter><!-- FIXME: Remove - do not commit, debugging for #24015 -->
+                  <artifact>commons-io:commons-io</artifact>
+                  <excludes>
+                    <exclude>org/apache/commons/io/input/BoundedInputStream*</exclude>
+                  </excludes>
+                </filter>
                 <filter>
                   <artifact>org.webjars.npm:tag2link</artifact>
                   <excludes>

[sebastic]

I tried to build the josm package with the commons-compress sources from josm-sources.jar, which requires libcommons-io-java & libcommons-lang3-java as these transitive dependencies are not included.

The build fails with those sources:

[gettext-extract] Warning: program compiled against libxml 212 using older 209
[gettext-extract] xgettext: Comment at or before ../src/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java:279 is not UTF-8 encoded.
[gettext-extract]           Please specify the source encoding through --from-code.
[gettext-extract] xgettext returned 1
  [antcall] Exiting /build/josm-0.0.svn19253+dfsg2/i18n/build.xml.
  [antcall] Exiting /build/josm-0.0.svn19253+dfsg2/i18n/build.xml.
      [ant] Exiting /build/josm-0.0.svn19253+dfsg2/i18n/build.xml.

BUILD FAILED
/build/josm-0.0.svn19253+dfsg2/build.xml:325: The following error occurred while executing this line:
/build/josm-0.0.svn19253+dfsg2/i18n/build.xml:126: The following error occurred while executing this line:
/build/josm-0.0.svn19253+dfsg2/i18n/build.xml:147: The following error occurred while executing this line:
/build/josm-0.0.svn19253+dfsg2/i18n/build.xml:91: Build failed

This is fixed by converting the file:

iconv -f ISO-8859-1 -t UTF-8 --verbose -o src/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java src/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java

But excluding the sources for the dependencies in i18n/build.xml seems like a better solution:

-            <fileset dir="${josm.build.dir}/src" includes="**/*.java"/>
+            <fileset dir="${josm.build.dir}/src/org/openstreetmap/josm" includes="**/*.java"/>

This seems to resolve the issue as commons-io.jar & commons-lang3.jar are now on the classpath for commons-compress.

Should these two transitive dependencies also be added to ivy.xml like xmcore for metadata-extractor? Then again, Jetbrains annotations aren't included for OpeningHoursParser either.

[taylor.smock]

TarArchiveOutputStream.java isn't really supposed to be part of our source code hierarchy. With that said, I'd probably do the following:

1. Add minimal dependencies for i18n/build.xml (what are you using this for anyway? The generated i18n files are part of the source tree)
2. Run whatever you need to in i18n/build.xml
3. Add remaining dependencies.

[sebastic]

Replying to taylor.smock:

TarArchiveOutputStream.java isn't really supposed to be part of our source code hierarchy.

That's why I prefer excluded those files from the xgettext fileset.

With that said, I'd probably do the following:

1. Add minimal dependencies for i18n/build.xml (what are you using this for anyway? The generated i18n files are part of the source tree)

"It was like that when I got here."

With the "generated i18n files" you mean nodist/trans/*.lang?

Those contains much less than what gets generated during the package build.

2. Run whatever you need to in i18n/build.xml
3. Add remaining dependencies.

I don't understand what you mean with this.

We just include the i18n/build.xml in our build.

The commons-io & common-lang3 dependencies are for commons-compress.

One of the source files of commons-compress from josm-sources.jar is not UTF-8 encoded, which causes xgettext to fail.

If the .lang files we generate contain a bunch of unused strings, we could drop all of the i18n customizations and just install nodist/trans/*.lang in the josm-l10n package.

[taylor.smock]

Replying to sebastic:

"It was like that when I got here."
With the "generated i18n files" you mean nodist/trans/*.lang?
Those contains much less than what gets generated during the package build.

The generated i18n files are nodist/trans/*.lang and resources/data/*.lang. I don't know what you are actually packaging (IIRC, the i18n files are split out), but something to keep in mind is that the *.lang files in the JOSM core repo are for strings in JOSM core only; the i18n build script will work on all plugins in JOSM svn, so that may be where the size discrepancy comes from. Plugins should (mostly) have their respective i18n files in version control, and also in the released jar files.

2. Run whatever you need to in i18n/build.xml
3. Add remaining dependencies.

I don't understand what you mean with this.

I am not that familiar with debian packaging, but I assume you can have multiple steps. In this case, extract JOSM source -> run i18n script -> extract non-JOSM sources -> build JOSM.

If the .lang files we generate contain a bunch of unused strings, we could drop all of the i18n customizations and just install nodist/trans/*.lang in the josm-l10n package.

This is almost certainly the case, unless you also distribute plugin jar files. But those should also have their respective i18n files in source control.

[sebastic]

The Debian package build just calls ant <various options> dist.

The i18n customizations likely have their origins in the need to have the preferred form of modification for the translations in the source package and to ensure they get built from source.

We used package plugins too (​josm-plugins), but that was too much of a PITA to keep around.

Now that the josm package is no longer using libcommons-compress-java (but only its dependencies), we can consider this issue resolved. The lack sources for transitive dependencies in josm-sources.jar is still a bit a pain, but it's good enough for now.

[sebastic]

Just for the record: The package build fails for the bookworm backport because that has commons-io & commons-lang3 2.11.0 and the commons-compress sources are 2.17.1.

We'll need to include the sources for the dependencies as well or revert back to the commons-compress package.