[PATCH] Introduce Apache IVY for dependency management

[wiktorn]

Before building JOSM you need to call ant bootstrap to download ivy. Use
bootstrap-workspace task to populate lib/ directory with all libraries
used by project. This is useful when working with IDE. Eclipse project
is migrated to use this files.

Open issues:

whether ant clean task should also clean ivy cache
whether include ivy jar in JOSM repository or use bootstrap task and
download it to ant home
whether to use version in names in lib/ directory
why some classes in org/apache/commons/jcs/auxiliary/remote/ are
required in runtime
what to do with kitfox svg and gnu getopt

You can review/comment patch at:
​https://github.com/wiktorn/josm/pull/2

Only first 4 files needs attention. Rest is removal of svn:externals source files and libraries from repository.
See: #8269

[michael2402]

Replying to wiktorn:

you need to call ant bootstrap to download ivy.

NPM uses 'npm install', but I think this will cause confusion for ant developers.

Use bootstrap-workspace task

Can we call that one eclipse (same as gradle does)
The corresponding clean task whould be cleanEclipse

Open issues:

whether ant clean task should also clean ivy cache

Usually, build tool download caches are excluded from clean. I don't know how well ivy handles dependency changes (e.g. new versions), but the tools I usually work with (maven, gradle, npm, composer) are fine with it and don't need the files cleaned if a library gets updated.

whether include ivy jar in JOSM repository or use bootstrap task and
download it to ant home

If you download it, you end up building your own package management software to download the package management software you want to use... :D. So I'm for just dropping the jar file in there.

whether to use version in names in lib/ directory

It would make work much easier.
Mind that you would need to auto-generate the classpath file then (or simply drop native eclipse support and use gradle for that)

why some classes in org/apache/commons/jcs/auxiliary/remote/ are
required in runtime
what to do with kitfox svg and gnu getopt

I simply put them into a different source set for builds - or what did you mean?

Some notes on the config:

• We should have a testRuntime. This is where you e.g. put the junit-jupiter-engine
• Your configuration setup seems weired. Usually, the runtime-Configuation extends the compile-Configuration, not the other way around.

In maven, compile (new gradle calls it implementation for non-transitive or api for transitive dependencies) is used for the dependencies that are required for your classes to link against. runtime are the additionaly dependencies that are only required during runtime (e.g. actual logger implementations, containers like tomcat, ...). They are detected using some reflection magic, this is why you don't need them when compiling ;-)

[stoecker]

Hmm, do I see this right, that it will externalize all resources?. Actually that's the opposite of what I'd consider a good idea. Rather than externalize all resources I'd like to have local copies of everything which we use as svn:external ATM.

It is simply too easy to compromise many external references.

An semi-automatic update process would be very fine, but no automatic one.

[wiktorn]

Replying to michael2402:

Replying to wiktorn:

Use bootstrap-workspace task

Can we call that one eclipse (same as gradle does)
The corresponding clean task whould be cleanEclipse

If we would generate project files then yes. Right now - it could be used by Eclipse, IntelliJ IDEA or any other IDE.

Open issues:

whether ant clean task should also clean ivy cache

Usually, build tool download caches are excluded from clean. I don't know how well ivy handles dependency changes (e.g. new versions), but the tools I usually work with (maven, gradle, npm, composer) are fine with it and don't need the files cleaned if a library gets updated.

It handles such situations quite well, the only problem is ever-growing ivy cache. But I lean towards not cleaning ivy cache.

whether include ivy jar in JOSM repository or use bootstrap task and
download it to ant home

If you download it, you end up building your own package management software to download the package management software you want to use... :D. So I'm for just dropping the jar file in there.

Ok. Then we can use Ivy to update itself :-)

whether to use version in names in lib/ directory

It would make work much easier.
Mind that you would need to auto-generate the classpath file then (or simply drop native eclipse support and use gradle for that)

I specially resigned from version in file names to avoid updating project files on each dependency update

what to do with kitfox svg and gnu getopt

I simply put them into a different source set for builds - or what did you mean?

Right now I kept them as they are. I was thinking about pushing our changes upstream and clean this up.

Some notes on the config:

• We should have a testRuntime. This is where you e.g. put the junit-jupiter-engine

And use different test configuration for compiling tests, and testRuntime for running tests. I'm not sure if I see any gains from this separation.

• Your configuration setup seems weired. Usually, the runtime-Configuation extends the compile-Configuration, not the other way around.

Indeed. But I think, that both runtime may have extra libraries not needed by compilation, as the compilation may need libraries not needed by runtime.

As I'm creating über-jar from runtime libraries I'd like to avoid putting in runtime configuration libraries, that are needed only for compilation to save space. Anyway generated jar has already grown 2MB.

[wiktorn]

Replying to stoecker:

Hmm, do I see this right, that it will externalize all resources?. Actually that's the opposite of what I'd consider a good idea. Rather than externalize all resources I'd like to have local copies of everything which we use as svn:external ATM.

Yes, it will externalize all resources. I consider local copies of everything a bad idea - it makes our repository grow really fast. Why would I trust JOSM repository more than official one? I know quite a few developers for whom this would be a showstopper due to security reasons ("they ship their own version of this library, that's fishy"). Especially when we know ship our jars (e.g. animalsniffer) that were patched by us and are not part of official release.

It is simply too easy to compromise many external references.

Usual reason for compromise is use of outdated dependencies (OWASP 2017:A9). I'd set a higher priority for easy update of dependencies than protection from compromise of official repositories.

An semi-automatic update process would be very fine, but no automatic one.

We could use ivy to download dependencies to specific folder (as it's done right now in bootstrap-workspace task) and commit that to repository.

Though I think we have better ​ alternatives. As we have already setup our Nexus, we could just remove other repositories from our ivysettings.xml and download only from ours and implement checksum verification there.

Though specifying checksum in dependency entry (as asked in the SO question) would be the best, provided that it is recursive hash (hashes through dependencies of dependency).

[michael2402]

Replying to wiktorn:

Replying to michael2402:

Replying to wiktorn:

Use bootstrap-workspace task

Can we call that one eclipse (same as gradle does)
The corresponding clean task whould be cleanEclipse

If we would generate project files then yes. Right now - it could be used by Eclipse, IntelliJ IDEA or any other IDE.

You should generate the project files. If someone does not want to import it as ant project but e.g. use the gradle nature instead there would always be conflicts in the classpath files. And it would mean that we still have to maintain/keep in sync the two lists of dependencies (the ones in the eclipse file and the ones in ivy).

whether include ivy jar in JOSM repository or use bootstrap task and
download it to ant home

If you download it, you end up building your own package management software to download the package management software you want to use... :D. So I'm for just dropping the jar file in there.

Ok. Then we can use Ivy to update itself :-)

I don't know how well ant+ivy is at this, but if that is possible it would be nice. I did not find the equivalent to the gradle ./gradlew wrapper --gradle-version 4.8.1.

what to do with kitfox svg and gnu getopt

I simply put them into a different source set for builds - or what did you mean?

Right now I kept them as they are. I was thinking about pushing our changes upstream and clean this up.

This would be a good idea. For getopt, dropping getopt might be a good idea. It is a very old (~40 years) C-style library that was just copy+pasted to Java and porting something upstream would mean to patch the C library. The source is just ugly.

Some notes on the config:

• We should have a testRuntime. This is where you e.g. put the junit-jupiter-engine

And use different test configuration for compiling tests, and testRuntime for running tests. I'm not sure if I see any gains from this separation.

• Your configuration setup seems weired. Usually, the runtime-Configuation extends the compile-Configuration, not the other way around.

Indeed. But I think, that both runtime may have extra libraries not needed by compilation, as the compilation may need libraries not needed by runtime.

Libraries that you expect to be provided by the environment (like if you compile a JOSM plugin that will run with the default JOSM libraries loaded) are added to the provided scope ;-)

Libraries that are required by the build system but not by the actual program (like ivy, ant, ...) are buildscript dependencies (at least in gradle. Maven only uses plugins)

Libraries that you only link against and that you do not need during runtime are a sign that something with your build system is broken. As for the com.github.spotbugs annotations, simply put them in the provided scope. That is not good practice but it works in our case, since those are only annotations. It would be better to find out if it is really used (I could not find a mention of findbugs or spotbugs in src) and drop it otherwise.

[wiktorn]

Replying to michael2402:

You should generate the project files. If someone does not want to import it as ant project but e.g. use the gradle nature instead there would always be conflicts in the classpath files. And it would mean that we still have to maintain/keep in sync the two lists of dependencies (the ones in the eclipse file and the ones in ivy).

I added ivynature to the project and removed all dependencies. Though I do not generate the project I guess that this will be sufficient.

I don't know how well ant+ivy is at this, but if that is possible it would be nice. I did not find the equivalent to the gradle ./gradlew wrapper --gradle-version 4.8.1.

I've just added that.

Libraries that you only link against and that you do not need during runtime are a sign that something with your build system is broken. As for the com.github.spotbugs annotations, simply put them in the provided scope. That is not good practice but it works in our case, since those are only annotations. It would be better to find out if it is really used (I could not find a mention of findbugs or spotbugs in src) and drop it otherwise.

I'd consider annotations more as a buildscript type of dependency. It's only needed during execution of build script (as well as javacc and other generators). And I'd like to keep the possibility to use these annotations in src, as we may use them to silence all false-positives and bring their number to 0.

So do I understand you correctly that because we ship uberjar all our runtime libs are at the same time provided libs? Should runtime extend provided configuration? And in our case it will be empty extension?

[michael2402]

You should define:
implementation: all libs JOSM uses that plugins should not use
api: all the other libs
runtimeOnly: The libs shipped with JOSM that we do not need during compilation
provided: The libs we need during compilation but not on application start
epsg: Everything epsg needs- which is currently all of JOSM.

Then use those classpaths:
Compiling: implementation + api + provided
Jar file: implementation +api + runtimeOnly
Running: same as jar file

[stoecker]

Replying to wiktorn:

Replying to stoecker:

Hmm, do I see this right, that it will externalize all resources?. Actually that's the opposite of what I'd consider a good idea. Rather than externalize all resources I'd like to have local copies of everything which we use as svn:external ATM.

Yes, it will externalize all resources. I consider local copies of everything a bad idea - it makes our repository grow really fast.

If I have to compare security and repository size, then I choose a bigger size. SVN repo currently is nearly 1GB. For a decade of active development I think that's ok.

Why would I trust JOSM repository more than official one?

Well. The users are using JOSM, not some library. WE are responsible for what we deliver, which means we also need to check what we are using. This is impossible when it is automatic. That check can be easy (I trust them, copy) or more complex (what's the diff) - That's decision of the developer (which can also be wrong BTW).

I know quite a few developers for whom this would be a showstopper due to security reasons ("they ship their own version of this library, that's fishy").

Hmm, then maybe these people should never be let near security related software. Preferring uncontrollable remote storage over local copy is strange. Like all these "We get offline when someone deletes a minor JavaScript package" website issues recently.

Especially when we know ship our jars (e.g. animalsniffer) that were patched by us and are not part of official release.

Our task is not to re-release libraries, but we are using them. Patching stuff only used by JOSM is perfectly fine, when needed. The only drawback is that patches make additional work when updating (which sometimes means no update is done for a long time). When somebody thinks this is "fishy", then he did not understand the concept of open source.

It is simply too easy to compromise many external references.

Usual reason for compromise is use of outdated dependencies (OWASP 2017:A9). I'd set a higher priority for easy update of dependencies than protection from compromise of official repositories.

There are so many reasons against automatic updates:

• freshly introduced upstream bugs
• incompatible API changes/software changes
• upstream breaks completely (e.g. looses interest)
• license changes
• upstream is hacked

Look at Linux distributions. They all have local copies of the files and don't rely on remote stored files. I'm responsible for hundreds of packages in openSUSE and I can ensure you, that upstream breaks, all the time, in all possible ways.

An semi-automatic update process would be very fine, but no automatic one.

We could use ivy to download dependencies to specific folder (as it's done right now in bootstrap-workspace task) and commit that to repository.

That sounds much better.

Though I think we have better ​ alternatives. As we have already setup our Nexus, we could just remove other repositories from our ivysettings.xml and download only from ours and implement checksum verification there.

Though specifying checksum in dependency entry (as asked in the SO question) would be the best, provided that it is recursive hash (hashes through dependencies of dependency).

That solves only one of the problems, namely the "hacked" one.

[wiktorn]

Replying to stoecker:

Replying to wiktorn:

I know quite a few developers for whom this would be a showstopper due to security reasons ("they ship their own version of this library, that's fishy").

Hmm, then maybe these people should never be let near security related software. Preferring uncontrollable remote storage over local copy is strange. Like all these "We get offline when someone deletes a minor JavaScript package" website issues recently.

I'd say - they prefer trusted source than untrusted (i.e. consider JOSM source repository untrusted source of Apache commons comparing to official Apache page) source of library. Local copy is only after I downloaded it from somewhere. And what this changes - is how do they get it.

There are so many reasons against automatic updates:

• freshly introduced upstream bugs
• incompatible API changes/software changes
• upstream breaks completely (e.g. looses interest)
• license changes
• upstream is hacked

I guess that I misunderstood you orignally. With Apache Ivy we have two possibilities to specify dependencies:

• static - e.g. rev="2.2"

  <dependency org="org.apache.commons" name="commons-jcs-core" rev="2.2" conf="runtime->default"/>
  

• dynamic - e.g. rev="latest.release"

  <dependency org="org.apache.ivy" name="ivy" rev="latest.release" conf="ivy->default" />
  

I'd say, that for all our runtime dependencies we should use static dependencies. For buildscripts and other tools I guess we could consider switching to dynamic. Apart from Ivy, all I specified all dependencies as static. It covers all your remarks but:

• upstream is hacked

Though I think we have better ​ alternatives. As we have already setup our Nexus, we could just remove other repositories from our ivysettings.xml and download only from ours and implement checksum verification there.

Though specifying checksum in dependency entry (as asked in the SO question) would be the best, provided that it is recursive hash (hashes through dependencies of dependency).

That solves only one of the problems, namely the "hacked" one.

As this is the only one, that we have :-)

[Don-vip]

Replying to wiktorn:

For buildscripts and other tools I guess we could consider switching to dynamic.

This won't work, regressions or incompabilities with newest Java versions for checkstyle, error_prone or jacoco are frequent, and I often needed to skip the ".0" release to wait for the ".1" fix (r12457), use a custom version (r13719) or patch it directly (r13305). We must control the version as well.

[michael2402]

Also mind that we should watch out for dependencies that use dynamic versions in their dependencies. In that case we need to pin those dependencies as well. In the java world this is - thankfully - not so common and I did not notice such a thing in the JOSM dependencies.

There is an other problem with not pinning build tool versions: Builds cannot easily be reproduced. We won't be able to completely do this (e.g. because everyone uses a different version of ant), but we should at least try to do our best ;-).

[anonymous]

Replying to michael2402:

You should define:
implementation: all libs JOSM uses that plugins should not use
api: all the other libs
runtimeOnly: The libs shipped with JOSM that we do not need during compilation
provided: The libs we need during compilation but not on application start
epsg: Everything epsg needs- which is currently all of JOSM.

Done, based on current plugins I divided dependencies into api and implementation. This may need further review.

For now I left runtime, compile etc. configurations. But this could be easily moved to ant file.

[Don-vip]

It's done through #16860