[Patch] Maven support

[simon04]

As many IDEs are able to import Maven projects without any configuration required, I started a Maven build file. My aim is to ease the start in JOSM development and not necessarily to replace the Ant build file.

At the moment, compiling and building the jar works. Directly executing JOSM (w/o jar) fails due to a classpath problem (the images aren't found).

[bastiK]

Replying to simon04:

As many IDEs are able to import Maven projects without any configuration required, I started a Maven build file.

For example?

[jttt]

Eclipse/Netbeans/IDEA can automatically import maven projects and maven can run ant scripts so it can be nicely integrated. We use maven a lot at work and it really did simplify working on new projects. On the other way, josm build system is pretty easy and only improvement I can see is that users of other IDE's than Eclipse will also have project available and ready to run.

[anonymous]

Using maven is great idea. It would be nice to have example pom for plugin template „00_plugin_dir_template”

[Don-vip]

I'd like to achieve that after Java 9 is released. For now some Maven plugins are not compatible (maven-shade-plugin for example, and everything that relies on ASM) so we will have to wait anyway. But we can prepare the "mavenization" by reorganizing the tree to standard Maven directories (src/main/java, etc.).

We could get rid of svn:externals for our dependencies and fetch them from Maven central, patching them if necessary using Maven patch plugin, it works nice.

This way the github mirror should not break as often it currently does.

[stoecker]

Actually I don't like that for several reasons:

• As software packager for openSUSE many maven based packages are a hell to build
• The suggested Build-Time dependencies of maven aren't compatible with build infrastructures like OBS
• We have a working build system ant
• I'm against trendy software which is switched all the time without real benefits.

I have no problem with additional support, but oppose a replacement.

[michael2402]

Replying to stoecker:

Actually I don't like that for several reasons:

• As software packager for openSUSE many maven based packages are a hell to build
• We have a working build system ant

As someone who has set up travis to work with JOSM, I can say the same about ant ;-).

But I don't really like maven - it is not so easy to add custom components and dependencies that don't use maven.

Some plugins are using gradle. I, too, did some experiments with it some time ago gradle.zip​ (build/tests working). The advantage would be that gradle has a wrapper included that automatically downloads all dependencies for building the application. So even new users only need to run a ./gradlew.

IDE support for both is mixed: As long as you don't do anything special, it works. But if you do anything non-standard (two projects using the same source, mixed Java versions, ...), it can confuse the IDE.

The only benefit for JOSM I see is that it would make our dependency management easier in the long run.

I would see a lot of improvement for plugins: We can remove the dependency on a given source layout and they can be hosted on github/... more easily if we have a better dependency management.

[stoecker]

I would see a lot of improvement for plugins: We can remove the dependency on a given source layout and they can be hosted on github/... more easily if we have a better dependency management.

Actually that's nothing I want. The current GitHub plugin group is already a compromise and it causes additional work. Each time for an i18n update there is something special about the Github plugins. One checkout does not accept the binary file diffs and needs a re-checkout. The other changed the settings and checkin is impossible, ... Beside the fact that instead of one checkin I need to checkin the files for all plugins individually.

And each of these already has a different release system which needs additional work compared to the SVN when I want to release a bug-fixed version. And then you need to take care of the individual and MANUAL version handling or the bug-fix will not be distributed.

So actually while it may have advantages for individual plugin authors from my point as maintainer it is unwanted. And on the long term my experience tells me that ALL plugins are unmaintained and I don't encourage anything which makes more work for this case.

So for me GitHub still is worse compared to the SVN. The only real advantage is the Pull-Request system there.

[Don-vip]

Replying to michael2402:

But I don't really like maven - it is not so easy to add custom components and dependencies that don't use maven.

Well I don't like Gradle neither...

So for now we all have our preferred build system :) I'll take this into account.

[bastiK]

I have yet to see a build system that beats Make.

[stoecker]

Replying to bastiK:

I have yet to see a build system that beats Make.

Actually that's also my opinion. All these systems have been developed to replace make. And all fail in my eyes. Yes, make is a bit strange in the beginning, but after you know it nothing is really better.

[bastiK]

The comment was slightly tongue-in-cheek, but there is some truth to it.

I've used Maven in one project and what I like about it, is how easy you can add and try out different libraries, that would otherwise take time to install correctly.

It is amazing to watch how it needs to download half the internet, after a tiny change to the build file or at first build. Maybe I'm too paranoid, but how do you trust the binaries downloaded from Sonatype's Maven Central? ​Apparently 3rd parties can build and upload artifacts of a library if the original developer team is not interested. An attacker only needs to compromise one binary in order to add backdoors to any software that depends on it. With svn externals, we know where the code is coming from and we build it ourselves. (I trust the developers with commit access to these repositories, but not necessarily the vetting process by Sonatype that allows practically anonymous binary uploads.)

From Michael's gradle setup, I've learned that we already have a Nexus repository: #12263 (quite impressive!). If I understand correctly, this mirrors all the required dependencies from Maven Central + adds the artifacts that we produce. Now this is a plus in terms of reliable access to build dependencies, but it doesn't change the fact that you have to trust the integrity of the binary .class files, that eventually get copied into josm.jar (or does it?).

[michael2402]

Replying to bastiK:

From Michael's gradle setup, I've learned that we already have a Nexus repository: #12263 (quite impressive!). If I understand correctly, this mirrors all the required dependencies from Maven Central + adds the artifacts that we produce. Now this is a plus in terms of reliable access to build dependencies, but it doesn't change the fact that you have to trust the integrity of the binary .class files, that eventually get copied into josm.jar (or does it?).

No, it does not.

But if we really want to check the versions ourselves, it would be best to always specify a fixed version. We can then add the hash of that artifact and ensure that it will always be the same (​for gradle, there is the same for Maven somewhere). It is not 100% perfect (what if someone compromises that plugin...).

I'm always open for changes, but only if we have a real advantage from it. So I'd prefer git if we really use it for branching. But it would be no use if we simply change it to have a more modern version control (... I have not really worked with SVN for years). Same for plugins. Wordpress uses a single SVN plugin repository that contains more than one million revisions, and it still works. The only issue is that trac does not like to display the root directory of that repo.

[michael2402]

While the build change is off the table, what about resources.

We currently have a lot of test data (residing in test/data, data_nodist)

This data is pretty unstructured right now. A lot of it is used for multiple tests, some of it only used for specific tests.

For the join areas test (org.openstreetmap.josm.actions.JoinAreasActionTest), I find test data in:

• data_nodist/Join_Areas_Tests.osm
• data_nodist/Join_Areas_Tests_joined.osm
• test/data/regress/10511/10511_mini.osm
• test/data/regress/11992/shapes.osm

This makes debugging the tests relatively hard.

I'd suggest to change this and move all test related files to a resources directory.
We could use:

• test/data/org/openstreetmap/josm/actions/JoinAreasActionTest/testExamples.osm
• test/data/org/openstreetmap/josm/actions/JoinAreasActionTest/testExamples_joined.osm
• test/data/org/openstreetmap/josm/actions/JoinAreasActionTest/testTicket10511.osm
• test/data/org/openstreetmap/josm/actions/JoinAreasActionTest/testTicket11992.osm

This would make finding the test files for a given test much easier and makes it easier to see, if a file is unused.

[stoecker]

For files used only for single tests that sounds good, but that leaves the multi-used files unstructured and also there must be a guideline what to do when a single-test file is useful for a second test. Also the directory structure use overly complex.

What about a reduced variant:

• test/data/JoinAreasActionTest/testExamples.osm
• test/data/more_than_one_test.osm

There hierarchy is much smaller and changing singletest to more than one test is simply done by moving the file one level up (or by creating a directory which names more than one test i.e. /test/data/FirstTest_SecondTest/file.osm).

[michael2402]

I personally don't have a problem with the deep hierarchy. We currently have 434 java unit tests. We have 160 files in the test/data directory and I expect them to grow if we use more functional tests. I'd rather have more directories than one directory full of hundreds of other directories. It does not scale well.

The main reason why I chose this format is to make it easy to get the test files without putting them all in the same directory: Simply use the full qualified test class name, replace '.' by '/' and you have the directory.

[Don-vip]

Replying to michael2402:

I personally don't have a problem with the deep hierarchy.

Me neither.

I'd rather have more directories than one directory full of hundreds of other directories. It does not scale well.

+1, the number of directories in test/data/regress can only grow forever.

[stoecker]

If you like the deep dirs, fine.

Still - what's the rule for multi-usage?

[Don-vip]

For personal reasons I won't have enough time to work on this subject.

[michael2402]

Can we switch the layout of the source packages using src/main/java, src/getopt/java, ...

There are multiple benefits from this:

• It is always bad to have your source polluted by external refs.
• It would make the ant build script cleaner
• Static code analysis (javadoc, checkstype, ...) could be restricted to 'our' files without the need of white/blacklists
• eclipse currently has problems if you register multiple source sets that contain the same directory - this makes it impossible to have eclipse-compatible build scripts that enforce dependency barriers between josm code and the external libraries included in the code.

Before we do this, it would probably be best to move to git. That way we can experiment on a branch without destroying the main repo. I don't know how good SVN works with patches when it comes to moving files.

[Don-vip]

Replying to michael2402:

Before we do this, it would probably be best to move to git.

We should discuss about this in a new ticket.

[bastiK]

Replying to michael2402:

Can we switch the layout of the source packages using src/main/java, src/getopt/java, ...

+1 on putting library sources in separate directories, but we should think about the particular directory layout.

It should be something that can be compiled as a multi-module Maven project (see e.g. ​this example) and at the same time as Java 9 modules (#15229, see ​github-project for a possible layout).

Before we do this, it would probably be best to move to git. That way we can experiment on a branch without destroying the main repo.

Subversion supports branches - it is quite stable, functional and possibly one of the most underrated features. For the move to Git I suggest reading the countless prior discussions. Basically the prerequisites are:

• Work out a concept for all the server processes that currently depend on svn version number, most notably matching of plugin versions to compatible JOSM version.
• Convince Dirk that this is a good idea
• Offer to do the bulk of the work (which will be plenty)

[michael2402]

Replying to bastiK:

Replying to michael2402:

Can we switch the layout of the source packages using src/main/java, src/getopt/java, ...

+1 on putting library sources in separate directories, but we should think about the particular directory layout.

It should be something that can be compiled as a multi-module Maven project (see e.g. ​this example) and at the same time as Java 9 modules (#15229, see ​github-project for a possible layout).

My Layout was inspired by the default gradle source layout, which is: src/<srcSet name>/<language>. Gradle supports multiple source sets per project. That's what I use for building, so this is more like a personal favorite of mine instead of a technical restriction ;-)

If we want to move to packages and use maven, a sub-project like might be better. Like this:

• /main
  • /src/main/java
  • /src/test/java
• /projection
  • /src/main/java
  • /src/main/tesources
  • /src/test/java
  • /src/test/tesources
• /projection
  • /src/main/java
  • /src/generated/java <- we can move generated files there, so that SVN / git excludes and clean are easier.

(or other package names you desire). This would be the case for gradle as well.

We could add a special project for integration tests. And special projects for the java files we later use for building (for compiling epsg, we currently depend on all of josm. This currently forces us to re-compile epsg for every changed java file)

[michael2402]

Gradle build scripts. Needs clean files. Use: ./gradlew run -Porg.openstreetmap.josm.noJavaFX=true

[bastiK]

Replying to michael2402:

If we want to move to packages and use maven, a sub-project like might be better. Like this:

• /main
  • /src/main/java
  • /src/test/java
• /projection
  • /src/main/java
  • /src/main/tesources
  • /src/test/java
  • /src/test/tesources
• /projection
  • /src/main/java
  • /src/generated/java <- we can move generated files there, so that SVN / git excludes and clean are easier.

(or other package names you desire). This would be the case for gradle as well.

We could add a special project for integration tests. And special projects for the java files we later use for building (for compiling epsg, we currently depend on all of josm. This currently forces us to re-compile epsg for every changed java file)

That's about what I have in mind. It allows to have both a source set for production source code (main) and for tests (test) for each sub-project/module. (I'm not saying we should actually switch to Maven or Gradle as main build tool. But it is still a good idea to adopt the common directory layout, if we split the sources anyway.)

[michael2402]

So, I was made to volunteer for an other year of GSoC mentoring. I'll be spending some time on JOSM development this summer and have time to do this transition.

I'm planning to spend two weeks in May on JOSM development - getting up to date and fixing a few bugs here and there, mostly in UI / rendering code.

In comparison to other projects, I think that JOSM is relatively difficult to set up. My personal favorite plan of "fixing" this would be:

• Convert the svn repository to a git repository
• Split the src directory: Move JOSM main to a directory, gettext to the next one and so on. Don't change source files - so leave projection / ... extractions for later if they require source changes.
• Change the build system to gradle. It is done 80%, but the 20% that are missing can be made a lot easier after switching to the package scheme
• Don't include the jar files in our repo any more
• Move the source to github - it has good source management, allows for easy external pull requests, reviews and branch management. Disable the 'issues' feature there.
• Don't develop on master any more - use branches, merge them when CI says it's ok.
• Integrate Trac with github
  • Just a normal git remote
• Let our CI tools use github.
  • Github login for Jenkins
  • Let Jenkins publish build status to github
  • Alternative: Use travis, it's free and easy to set up (just give it the gradle targets, it will tell you if everything is ok)
• Sonar Server: Use the github repo monitoring the master branch.
  • Use the ​github plugin for new PRs to automatically inform users about their Sonar issues in new code.
  • Use github accounts for login (​plugin)

A lot of this (moving files, generating the git repo) can be done by a script.

This is just my personal optinion. Before spending time on the topic I would like to hear yours and if it is even desired to change anything.

[stoecker]

Like proposed it will not work. I'll give a few reasons, there are many more.

• The plan in general violates JOSMs development concept: We don't do breaking changes. We have a continuous development.
• A switch to gradle may be possible, but it would require a parallel usage time together with ant of at least a year in which gradle can prove it is really better than what we have (which is not easy BTW - While I dislike ant my experience as RPM packager with gradle is much worse).
• Nobody convinced me until now that git is really better than SVN for JOSM. Yes it has advantages, but it also has disadvantages. Which is the big feature we need to spend months of work for switching?
• We wont switch to GitHub! JOSM ATM has full control over its source and everything related. A switch to a "source code provider" is out of question.
• There is a reason why we don't use branches, namely the continuous development. Like for Git: Branches have advantages, but also disadvantages. Success of JOSM proves that current concept works. There must be a reason to switch it.

What you propose is spending months of work (not only yours, but also that of Vincent and me - JOSM has roughly 5000 lines of code and many many configurations settings working under the hood which you don't know) to switch a working solution to something else. Why? We have heard many requests to switch to git in the past, but until now I did not hear a single argument why we should do this beside: I like it more.

What is so fundamentally broken with the current system, that it must be overthrown with something nobody knows if it will work as well or not? JOSM is an exceptionally successful project which I couldn't have hoped to be so. So it already does many things right. I'm not totally opposed to switching either the build system or the version control software or any other component. We already did major reworks in the past. But there must be reasons for this.

Conclusion: A GSOC project to change version control or build system makes no sense. There are other topics which are much more interesting. One of the most recent probably: What do we do on Windows after Java 11 if WebStart and other components vanish. That can involve major changes as well, but at least clearly targets an obvious improvement.

[Don-vip]

My two cents:

• SVN -> GIT: I would support this migration for these reasons:
  • the ability of squashing commits is useful to avoid the traditional commit we all do less than 30 seconds after we see we made a mistake
  • the ability of stashing work in progress is also very useful when we work on several things in parallel (like I'm doing right now with projections stuff...)
  • students don't learn SVN anymore at school. So in the long run it will become more and more difficult to attract new coders

But (there is always a but): don't forget JOSM auto-update is based on the SVN revision number! A git migration must not break this (solution to find)

• split the src: I approve. The Maven layout (now adopted by Gradle) is standard in Java, many tools are preconfigured for it, and all Java developers are used to it.

• build system: I disapprove "changing" it. As Dirk said, ant must remain functional, it would break the compilation of many people and many Linux maintainers. But it would be perfectly fine to have several build systems, with a common properties file. I personally don't like Gradle much and would prefer Maven, but as you propose to do the job, and I couldn't find the time to implement Maven, I wouldn't oppose a solution based only on Gradle and Ant.

• don't include jar files: I approve. It would be natively done with Maven and Gradle. We can setup Ivy for Ant.

• Move the source to Github: No. We've seen with Oracle decision to drop WebStart and JavaFX how it can be dangerous to trust a private company. But I understand your concern. Github is really great to review pull requests. Instead of migrating to Github, I would agree on the principle of moving from Trac to something else (Gitlab, Tuleap?), but this would require an insane amount of work (we use Trac for everything!)

• develop on branches: if we move to git it will be natural. With SVN it doesn't make much sense for JOSM.

• trac + (local) git: no problem

• Github/SonarQube login for Jenkins: no. We should find how to reuse Trac logins.

Otherwise I feel a bit alone on the Java side... I'd really like some help on #14120 (open for more than a year), #16082, #16129, etc.

So in summary I'd say:

• OK to move to git (local repo on our server) after we find how to preserve JOSM autoupdate
• OK to rework source tree
• OK to add Gradle in addition to Ant, and add Ivy to avoid keeping dependencies and tools in our repository
• OK to use branches once we switch to git
• OK to find how to reuse Trac credentials in Jenkins and Sonar

[wiktorn]

I strongly support moving to git. Preparing first patch with change of imagery caching was a major PITA as I was working only with patches and svn. I use git as a primary development repository and only I'm done - I commit to SVN. I love git for the ability to commit frequently and ability to rewrite history.

I also don't have a problem with moving to GitHub - many other OpenStreetMap projects are there. Using GitHub for pull requests and review and not for issues limits the risk that we take. All important things are either in issues or in commit messages which are not (only) on GitHub. We can use local git repo to integrate with trac and sync that with GitHub. We can then make code reviews on GitHub :-)

Developing on branches: +1 (once we are on git)

Github/SonarQube to login for Jenkins - maybe we could use OpenStreetMap OAuth provider instead?

Moving to maven/gradle - I'm netural.

As for JOSM version number - I guess we can replace it with build timestamp + commit id. I guess we can't rely on timestamp of commits as they may land in master with date far from past. This would work good for "latest". For "tested" maybe we could release them under the version same as milestone.

[floscher]

I'm also supporting a move to git, when I have to work with SVM repositories I always prepare commits in a git-clone and commit using git-svn. Conflict resolution is so much easier and I can have local branches, as soon as you do more complicated changes, git simply can handle them better.

Regarding the repository hosting: I have no strong opinion, if it should be GitHub, GitLab (self-hosted or gitlab.com) or something else.

With the build systems I'm in favor of Gradle. I could help improving the current Gradle build. Maybe also existing components (translation toolchain?) of the gradle-josm-plugin can be reused for the main Gradle build.
An advantage can also be, that you can write your build script in JVM languages like Java, Groovy and write unit tests for the build, so you are confident everything works as expected.

With versioning you should use git-tags (ideally with semantic versioning) and use git describe --long. Until the migration to git is complete, you could use git rev-list --count HEAD which should give you the number of commits so far.

[stoecker]

As said GitHub is no option. I do software development a bit longer than most of you and I wont go back to a proprietary company based solution when I already have a solution which works without these.

Regarding Git. Vincent - the features you name are indeed fine. Still they aren't enough for me to increase the complexity.

As the comments show it is already possibly to work with Git. There is the Git mirror for this purpose. The question is why should the JOSM central repository switch? Additional complexity means that someone must care for this. I know too many people struggling already with the concept of SVN. Git is way more complex. That people have trouble with svn, but not git is no argument to me: Someone how is not able to understand the difference between git and svn will never successfully contribute to JOSM which is way more complex.

The JOSM development concept is designed for the case that people spend very little effort and we still have a recent release. Vincent, if you remember you wanted to have proper releases with announcements and changelogs and so on. I said that's fine, but must be optional. Most of these things died down again, except the more detailed changelog. The proposal of branch usage and git again are diametral to the development concept to keep it simple. I've seen too many projects developing nice new stuff but never get it released and then they died.

The one-branch SVN is not only laziness or stubbornness from my side, but it also enforces the continuous development. Everybody is forced to care for the recent state. Nobody can live in his little branch/fork box for long.

I see that many of you fancy cool features, but probably not many of you have seen projects running full speed into a wall because development model did not fit. Current situation with Vincent being there and doing lots of work is extremely fine, but this has not been the case always and it probably also will not be the case always. People come and go. The core people of JOSM changed many times already because JOSM is a hobby and no job. JOSM survived that mainly because of the development model. I don't see that a switch to git will improve the situation, I rather see the opposite.

If I look at some of the projects which have been in OSM SVN before and are on GitHub now I have the feeling dying already started. E.g. Tirex - Many different people contributed in SVN. Now on GitHub they did need months to handle a build fix where I provided the solution.

So please leave the fancy cool new features aside and check if a switch to git is NEEDED. What thing which is necessary can you do with git, which the current system cannot. I switch from RCS to SCCS to CVS to SVN (and also to git). Each of these steps removed MAJOR obstacles. What issue do we have with SVN now?

GitHub - many other OpenStreetMap projects are there

Well. Most OpenSource projects also have been on SourceForge and then the company got sold to somebody else.

don't include jar files: I approve. It would be natively done with Maven and Gradle. We can setup Ivy for Ant.

Right, that stuff does not belong into the repository, but no dynamic dependency handling instead. This is a

• security nightmare
• RPM maintainers nightmare. Did you every try to build such software without a network connection (like on most build servers)?

Already the externals in SVN are not good in terms of security and should be replaced with something non-automatic.

The other topics like source code reorganization, additional build systems, cleanups, ...: If it improves situation: Fine. Only to make it different: Please not.

[wiktorn]

Replying to stoecker:

Regarding Git. Vincent - the features you name are indeed fine. Still they aren't enough for me to increase the complexity.

So please leave the fancy cool new features aside and check if a switch to git is NEEDED. What thing which is necessary can you do with git, which the current system cannot. I switch from RCS to SCCS to CVS to SVN (and also to git). Each of these steps removed MAJOR obstacles. What issue do we have with SVN now?

I went maybe shorter path - from CVS to SVN to GIT and also see a lot of improvements on each step. I just do not understand, why do you think that git model increases complexity? I guess, that it comes slowly time, that to develop JOSM you need to know two tools - GIT, as this is your usual SCM, and SVN, because this is what JOSM project choose.

Using GIT we can still support the same development model as we have with SVN - only one branch on "main" repository.

As for the issues I have with svn I have following:

• central model - I need to setup separate repository and sync it with git, to have private branches etc.
• need to be online to commit - when developing I often stumble on some minor improvements which are not directly with the feature I'm working on, that I could commit & push.
• meaning smaller commits out of bigger work is extra work that I usually skip and commit all changes as one bigger patchset
• because I'm syncing with svn - all my commits to SVN are not properly recognized as the same change by GIT - so if I do not merge with JOSM GIT mirror immediately, I'll receive a conflict later. Because there is some latency between commit to SVN, and when commit is available in GIT JOSM mirror, this is something that is easy to miss
• GIT <-> SVN mirror scripts fail from time to time
• no easy way to separate my bugix work from feature work (at least - using only SVN)

Nobody can live in his little branch/fork box for long.

Thanks to GIT I'm sitting in my little branch for ~1.5 year and merging upstream changes is not that much of fuzz. If I would have to stick only to SVN I'd abandon implementing this feature long time ago. I'm with my both hands for keeping code submissions to JOSM easy.

[michael2402]

Replying to Don-vip:

My two cents:

• SVN -> GIT: I would support this migration for these reasons:
  • ...

One more reason: SVN is not installed on many systems any more. GIT has evolved to be the standard tool to be supported by IDEs - most potential developers will have it installed

But (there is always a but): don't forget JOSM auto-update is based on the SVN revision number! A git migration must not break this (solution to find)

I see two use cases for this:

• Plugin compatibility requires comparable version numbers => Change the version numbers to be 18.4.0, 18.5.0, 18.5.1, 18.5.1-SNAPSHOT. Imo, we don't need compatibility checks in the nightly versions - we don't have a 'tested up to' flag and nightly testers are expected to update their installation every few days.
• Nightly updating requires check if installed version is current => For snapshot versions (and releases, but it can be ignored there), include the git commit hash and the branch name.

• build system: I disapprove "changing" it. As Dirk said, ant must remain functional, it would break the compilation of many people and many Linux maintainers. But it would be perfectly fine to have several build systems, with a common properties file. I personally don't like Gradle much and would prefer Maven, but as you propose to do the job, and I couldn't find the time to implement Maven, I wouldn't oppose a solution based only on Gradle and Ant.

I have used both for complex projects (and written own plugins for both of them). I wanted to use gradle for several reasons:

• Maven is realy verbose when it comes to complex setups. A task definition that takes 3 lines in gradle usually takes ~30 in maven and is more complicated to read.
• Maven is meant to pull dependencies, compile a single jar and deploy it. It can do this really well (probably better as gradle). But changing this flow and e.g. adding custom build steps is difficult.

• don't include jar files: I approve. It would be natively done with Maven and Gradle. We can setup Ivy for Ant.

• Move the source to Github: No. We've seen with Oracle decision to drop WebStart and JavaFX how it can be dangerous to trust a private company. But I understand your concern. Github is really great to review pull requests. Instead of migrating to Github, I would agree on the principle of moving from Trac to something else (Gitlab, Tuleap?), but this would require an insane amount of work (we use Trac for everything!)

I have experience with setting up gitlab. But the move from trac to <whatever ticket system> is more complicated - the wiki would need to move and that would be a lot of work.

• trac + (local) git: no problem

Trac won't care where your git origin is. If github dies, you just have to set up / move to a new git server somewhere. Changing the origin URL is not much work in git (just one command on the command line) - it's not SVN ;-)

• Github/SonarQube login for Jenkins: no. We should find how to reuse Trac logins.

... or OSM logins. Or move all JOSM logins to OSM. But I just think that you should be able to log in there using some OAuth service and not use more custom accounts.

Replying to Don-vip:

• build system: I disapprove "changing" it. As Dirk said, ant must remain functional, it would break the compilation of many people and many Linux maintainers. But it would be perfectly fine to have several build systems, with a common properties file. I personally don't like Gradle much and would prefer Maven, but as you propose to do the job, and I couldn't find the time to implement Maven, I wouldn't oppose a solution based only on Gradle and Ant.

I don't want to maintain two build systems. We either switch or we don't.

Linux maintainers will have more work mirgating from SVN to GIT than to change the 'ant xxx' line by a './gradlew xxx' line. They might even be happy to drop some dependencies. They might even be ​happy about it

Replying to stoecker:

As said GitHub is no option. I do software development a bit longer than most of you and I wont go back to a proprietary company based solution when I already have a solution which works without these.

Should not be a problem since we use git. We already use github as a project mirror. Many people developing JOSM use it as a remote. If github dies, it's a 5 second job to get the repo somewhere else. Git is much easier than SVN here (thanks to the decentralized Blockchain, to use some buzzwords :D). But that's the reason why I would not migrate issues / wiki there.

Someone how is not able to understand the difference between git and svn will never successfully contribute to JOSM which is way more complex.

For me, SVN is more complex. I used both (a lot) and have done SVN->GIT migrations (with branches, commit rewrites, ...). From a user perspective, GIT is much more intuitive. You have way better UI support for branching / pushing / ..., therefore it is much easier for new users to use. Have you ever configured a svn:exclide? Compare this to adding the files to .gitignore - way less magic, way easier to track.

The one-branch SVN is not only laziness or stubbornness from my side, but it also enforces the continuous development. Everybody is forced to care for the recent state. Nobody can live in his little branch/fork box for long.

This would be a good point. But in pracitce, it won't work. Have a look at ​my 280 branches. I can't even tell which one of those branches I merged back to main, which ones are added to some JOSM ticket as .patch file because they are not ready, which ones just did not work and which ones I forgot.

All other comments:

... anti-svn-arguments ...

One of the reasons why development was so hard for me. The computer I'm writing from just now does not have SVN installed. And no git as well.

The last time I did some JOSM development (some color changer bug) I spend 3 hours setting up build environment and one hour fixing stuff - and this was with a Computer that previously had JOSM installed. From feedback by the the GSoC students, setting up for core deveopment is not smooth and requires some help. This is the main reason why I would like to change to a more modern approach that requires less configuration.

[stoecker]

Using GIT we can still support the same development model as we have with SVN - only one branch on "main" repository.

That's very unlikely. Once we go to git that option is gone. Otherwise switching to git would be useless.

• central model - I need to setup separate repository and sync it with git, to have private branches etc.

Which is partly wanted :-)

• need to be online to commit - when developing I often stumble on some minor improvements which are not directly with the feature I'm working on, that I could commit & push.
• meaning smaller commits out of bigger work is extra work that I usually skip and commit all changes as one bigger patchset
• because I'm syncing with svn - all my commits to SVN are not properly recognized as the same change by GIT - so if I do not merge with JOSM GIT mirror immediately, I'll receive a conflict later. Because there is some latency between commit to SVN, and when commit is available in GIT JOSM mirror, this is something that is easy to miss
• GIT <-> SVN mirror scripts fail from time to time

We can have the git mirror on JOSM server. It can automatically sync. I started the setup of a GitHub JOSM mirror during the last sync outage, but I'd need someone with much more experience to help here to get a satisfying solution.

• no easy way to separate my bugix work from feature work (at least - using only SVN)

Yes. I'm aware of that. I consider the local commits one of the biggest features which could convince me.

Nobody can live in his little branch/fork box for long.

Thanks to GIT I'm sitting in my little branch for ~1.5 year and merging upstream changes is not that much of fuzz.
If I would have to stick only to SVN I'd abandon implementing this feature long time ago.

What are you talking about here?

P.S.:
After reading a bit more about git svn. A lot of the features request are already there. E.g. dcommit allows to commit multiple changes with one revsion per change and thus keeping the revision history.

What are the major troubles using git directly with the SVN?

[stoecker]

Trac won't care where your git origin is. If github dies, you just have to set up / move to a new git server somewhere. Changing the origin URL is not much work in git (just one command on the command line) - it's not SVN ;-)

Please keep discussion serious. svn relocate exists as well and a new "svn checkout" is much easier than doing the same for git. Lots of git features discussed aren't "git specific".

Changing a repository is not some minutes work. In no case.

Linux maintainers will have more work mirgating from SVN to GIT than to change the 'ant xxx' line by a './gradlew xxx' line. They might even be happy to drop some dependencies. They might even be ​happy about it

I doubt. I dropped RPM ports of a handful programs because of these dynamic dependencies and the brand-new-build tools which are never available in the required version for the target OS. And I worked with many many build systems because it seems nowadays each tool has its own.

From a user perspective, GIT is much more intuitive. You have way better UI support for branching / pushing / ..., therefore it is much easier for new users to use.

Sorry, but this I highly doubt. It's probably because you had this in your studies and SVN not. But calling Git intuitive is strange. DVCS maybe anything, but not intuitive. All experience with non-computer science people I have shows otherwise.

This would be a good point. But in pracitce, it won't work. Have a look at ​my 280 branches. I can't even tell which one of those branches I merged back to main, which ones are added to some JOSM ticket as .patch file because they are not ready, which ones just did not work and which ones I forgot.

This exactly indicates what I mean. Not even you yourself know what you did. For the JOSM SVN there exists only 1 version. There is no doubt about its state.

The last time I did some JOSM development (some color changer bug) I spend 3 hours setting up build environment and one hour fixing stuff - and this was with a Computer that previously had JOSM installed. From feedback by the the GSoC students, setting up for core deveopment is not smooth and requires some help. This is the main reason why I would like to change to a more modern approach that requires less configuration.

Hmm. What do you do? Install ant, svn, java-devel. svn checkout. ant. If that's too complicated it wont get better with maven, gradle, git and whatever. For me it is additionally disable JavaFX, because JavaFX is not available for my distribution.

That git is installed, but svn not is no argument for me. Sorry, but if people aren't able to install svn I don't consider them helpful at all. Like these many university based one-shot plugin projects we had. I don't want these - we don't need yet another plugin nobody cares about. JOSM very likely is not the right starting point for programming beginners. It is much too complex for this.

BTW: The number of contributions we get shows that people can compile JOSM. Most of them without every asking a single question first. The argument, that current system isn't acceptable does not convince me a single bit.

If we go to git it must have a real benefit, no pseudo arguments.

What I see as advantages with git for JOSM are:

• local commits (i.e. better history tracking) → offline work much easier
• better conflict resolution
• offline local history (which OTOH is also a drawback because of the size)
• I'm not sure about these, but Vincents history tuning features probably as well

[Don-vip]

Replying to stoecker:

• I'm not sure about these, but Vincents history tuning features probably as well

The idea about this is to hide some commits from the final history by merging them with a similar one:

• r13641 + r13642
• r13619 + r13620
• r13612 + r13614
• r13604 + r13605

These commits belong with the commit done a few minutes/hours before. By rewriting the history, it's easier to follow/understand for the other developers.

[wiktorn]

Replying to stoecker:

Nobody can live in his little branch/fork box for long.

Thanks to GIT I'm sitting in my little branch for ~1.5 year and merging upstream changes is not that much of fuzz.
If I would have to stick only to SVN I'd abandon implementing this feature long time ago.

What are you talking about here?

I have a branch with Imagery Definitions refactor. You can see here, that my work stared in October 2016:
​https://github.com/wiktorn/josm/compare/mirror...wiktorn:img_def_refactor

And I'm still able to continue development and merge upstream changes. With bare svn this would be impossible

P.S.:
After reading a bit more about git svn. A lot of the features request are already there. E.g. dcommit allows to commit multiple changes with one revsion per change and thus keeping the revision history.

What are the major troubles using git directly with the SVN?

If this will maintain link between git commit and svn commit then probably most of my arguments will go away. After short reading I'm really keen to try it on.

But if this is recommended way to work on JOSM, why SVN at all? What's the difference between:
apt install svn
And
apt install git
?

And you really don't need to explain more about git than about svn to get people unfamiliar with it to get it going.

• offline local history (which OTOH is also a drawback because of the size)

Currently it looks more less like this:

$ du -hs git/josm svn/josm
828M    git/josm
540M    svn/josm

So git repository is not that much bigger than svn checkout.

[floscher]

Replying to stoecker:

P.S.:
After reading a bit more about git svn. A lot of the features request are already there. E.g. dcommit allows to commit multiple changes with one revsion per change and thus keeping the revision history.

What are the major troubles using git directly with the SVN?

My main issue is, that you can't just clone the git-repo and then do git svn dcommit. You have to initialize git-svn, which normally requires to go through all thirteenthousandandsomething commits and because this is painfully slow for SVN (over the network, one commit at a time) this takes very long. There are ways to skip that, but that again requires a pretty good understanding on how git internals work. This adds additional complexity that would not be needed when the main repo was in git.

Another thing is, that svn:externals and svn:ignore are not handled by git-svn. You can show them and edit them, but they are not replicated to git. The externals are normally mirrored to git, but only for selected versions, so you can only build exactly those, not the ones in between. Ignores have to be curated separately for SVN and git and ​as we learnt recently, can behave strangely.

Replying to stoecker:

don't include jar files: I approve. It would be natively done with Maven and Gradle. We can setup Ivy for Ant.

Right, that stuff does not belong into the repository, but no dynamic dependency handling instead. This is a

• security nightmare
• RPM maintainers nightmare. Did you every try to build such software without a network connection (like on most build servers)?

Right, these are problems Maven/Gradle would introduce in the default configuration. But you can lock down your dependencies (also transitive dependencies) to a fixed version, so you always get the same dependency versions.
And regarding the offline requirement: Sure, you will need network access at some point to get the dependencies. But you can separate downloading of dependencies from building. Right when downloading the JOSM sources you also download the dependencies. Then you can later build offline using ./gradlew --offline build.

[anonymous]

Replying to stoecker:

Changing a repository is not some minutes work. In no case.

Setting up new project on local gitlab: less than a minute.

Change origin remote: less than 10 Seconds.

For Josm there wiuld be the initial push, but there is no real work involved there

This would be a good point. But in pracitce, it won't work. Have a look at ​my 280 branches. I can't even tell which one of those branches I merged back to main, which ones are added to some JOSM ticket as .patch file because they are not ready, which ones just did not work and which ones I forgot.

This exactly indicates what I mean. Not even you yourself know what you did. For the JOSM SVN there exists only 1 version. There is no doubt about its state.

The problem is that there are 2 Versions of JOSM - one in my git, one on master. Patches just got copy+pasted with no way to automatically reference them.

The last time I did some JOSM development (some color changer bug) I spend 3 hours setting up build environment and one hour fixing stuff - and this was with a Computer that previously had JOSM installed. From feedback by the the GSoC students, setting up for core deveopment is not smooth and requires some help. This is the main reason why I would like to change to a more modern approach that requires less configuration.

Hmm. What do you do? Install ant, svn, java-devel. svn checkout. ant. If that's too complicated it wont get better with maven, gradle, git and whatever. For me it is additionally disable JavaFX, because JavaFX is not available for my distribution.

I don't use Ant, I use eclipse as a build system. They are not in sync and eclipse is missing some config. My Patches for that are more than a year old and won't apply nicely. I did not get it to work. I then imported from gradle as workaround.

That git is installed, but svn not is no argument for me.

For me it is when doing quick fixes.

[Don-vip]

Replying to anonyme:

My Patches for that are more than a year old and won't apply nicely.

Who's speaking? It you can link the corresponding JOSM ticket I can take a look. I don't remember to have forgotten a proposed Eclipse patch.

[michael2402]

Replying to Don-vip:

Replying to anonyme:

My Patches for that are more than a year old and won't apply nicely.

Who's speaking? It you can link the corresponding JOSM ticket I can take a look. I don't remember to have forgotten a proposed Eclipse patch.

That was me. I did not submit ​patches like this one because it would make other builds fail. Or ​this one ;-).

And since you use Jenkins on master after committing, you do not need a ​travis config there. But I prefer to have the tests run before merge. And I prefer to test in forked mode since this catches the test errors at the right test - but it would decrease Performance on the Jenkins server and you don't gain much.

That's a bit more than just downloading ant + svn + checkout, since I prefer to work in a dev environment that provides me with some tools (Checkstyle, Unit tests, ..) and to hopefully test the stuff before pushing it to master - which is quite difficult at the current point.

[stoecker]

Replying to Don-vip:

Replying to stoecker:

• I'm not sure about these, but Vincents history tuning features probably as well

The idea about this is to hide some commits from the final history by merging them with a similar one:

• r13641 + r13642
• r13619 + r13620
• r13612 + r13614
• r13604 + r13605

These commits belong with the commit done a few minutes/hours before. By rewriting the history, it's easier to follow/understand for the other developers.

I know the why. I'm not sure about the if :-) It's like the submit comment changing. History tuning is sometimes dangerous (remember 1984 by Orwell).

[stoecker]

Replying to floscher:

Replying to stoecker:

P.S.:
After reading a bit more about git svn. A lot of the features request are already there. E.g. dcommit allows to commit multiple changes with one revsion per change and thus keeping the revision history.

What are the major troubles using git directly with the SVN?

My main issue is, that you can't just clone the git-repo and then do git svn dcommit. You have to initialize git-svn, which normally requires to go through all thirteenthousandandsomething commits and because this is painfully slow for SVN (over the network, one commit at a time) this takes very long. There are ways to skip that, but that again requires a pretty good understanding on how git internals work. This adds additional complexity that would not be needed when the main repo was in git.

As far as I understood a local mirror probably would solve this as it contains the necessary git-svn data for syncing which GitHub has not.

Another thing is, that svn:externals and svn:ignore are not handled by git-svn. You can show them and edit them, but they are not replicated to git. The externals are normally mirrored to git, but only for selected versions, so you can only build exactly those, not the ones in between. Ignores have to be curated separately for SVN and git and ​as we learnt recently, can behave strangely.

Right. But that's a small nuisance.

Right, these are problems Maven/Gradle would introduce in the default configuration. But you can lock down your dependencies (also transitive dependencies) to a fixed version, so you always get the same dependency versions.

I don't like that at all. As said I'm already ready to revert my decisions to use external SVN access. I don't think introducing another remote reference is an option. The world changed a bit in the last years and the security effects of these approaches show their face more often now.

And regarding the offline requirement: Sure, you will need network access at some point to get the dependencies. But you can separate downloading of dependencies from building. Right when downloading the JOSM sources you also download the dependencies. Then you can later build offline using ./gradlew --offline build.

That's good. I've had build systems where it is impossible, even with tricks like filling the temporary folder, to build offline.

[stoecker]

Replying to michael2402:

That's a bit more than just downloading ant + svn + checkout, since I prefer to work in a dev environment that provides me with some tools (Checkstyle, Unit tests, ..) and to hopefully test the stuff before pushing it to master - which is quite difficult at the current point.

But all that is not required. It's optional.

[wiktorn]

I keep in mind don-vip request to keep parallel ant and gradle builds.

don-vip proposed to use Apache Ivy to specify dependencies. I guess that I can give it a try.

@michael2402:
What do you think about following:
​https://github.com/smoothed9/Ivy-to-Gradle/blob/master/IvyToGradle.gradle

Would it make it easier, to keep both build systems in sync?

[michael2402]

Replying to wiktorn:

I keep in mind don-vip request to keep parallel ant and gradle builds.

I know. I won't be supporting this, since we would then be maintaining 3 build systems (ant, eclipse, gradle). The two we have at the moment are out of sync often enough.

don-vip proposed to use Apache Ivy to specify dependencies. I guess that I can give it a try.

@michael2402:
What do you think about following:
​https://github.com/smoothed9/Ivy-to-Gradle/blob/master/IvyToGradle.gradle

Would it make it easier, to keep both build systems in sync?

This converts a ivy file to a gradle file.

This is not even neccessary. Gradle build files can be scripted. You can write a converter that includes the dependencies from the ivy.xml file.

Those are the current gradle build script dependencies:

    classpath group: 'net.java.dev.javacc', name: 'javacc', version: '7.0.3'

Those are the current gradle dependencies:

    implementation group: 'com.drewnoakes', name: 'metadata-extractor', version: '2.11.0'
    implementation group: 'oauth.signpost', name: 'signpost-core', version: '1.2.1.2'
    api group: 'org.apache.commons', name: 'commons-jcs-core', version: '2.1'
    api group: 'commons-logging', name: 'commons-logging', version: '1.2'
    api group: 'org.apache.commons', name: 'commons-compress', version: '1.12'
    api group: 'javax.json', name: 'javax.json-api', version: '1.1.2'
    implementation group: 'org.glassfish', name: 'javax.json', version: '1.1.2'
    if (!project.hasProperty('org.openstreetmap.josm.noJavaFX')) {
    	// TODO JavaFx
    }
    implementation sourceSets.svgSalamander.output
    implementation sourceSets.getopt.output
    api group: 'org.openstreetmap.jmapviewer', name: 'jmapviewer', version: '2.6', ext: 'jar'

    checkstyleImplementation group: 'com.puppycrawl.tools', name: 'checkstyle', version: '8.8'
    
    // Scripts currently depend on all of JOSM - TOOD: Fix this.
    scriptsImplementation sourceSets.main.output
    
	testImplementation 'junit:junit:4.12'
	testImplementation "com.github.stefanbirkner:system-rules:1.16.0"
	testImplementation group: 'nl.jqno.equalsverifier', name: 'equalsverifier', version: '2.1.6'
	testImplementation group: 'com.google.code.findbugs', name: 'annotations', version: '3.0.1u2'
	testImplementation group: 'org.easytesting', name: 'fest-reflect', version: '1.4.1'
	testImplementation group: 'org.reflections', name: 'reflections', version: '0.9.10'
	testImplementation group: 'org.unitils', name: 'unitils-core', version: '3.4.3'
	testImplementation group: 'com.github.tomakehurst', name: 'wiremock', version: '2.12.0'
	testImplementation group: 'net.trajano.commons', name: 'commons-testing', version: '2.1.0'
	testImplementation group: 'org.awaitility', name: 'awaitility', version: '3.0.0'
	
	testFunctionalCompile sourceSets.test.runtimeClasspath
	testPerformanceCompile sourceSets.test.runtimeClasspath

If they (or at least the ones that are not svn:externals) get added to an ivy.xml file, I can simply read the information from there.

I don't have much experience with ivy, but it seems to be pretty well integrated into gradle.

The main problem with keeping them in sync so far was that all code in JOSM is just dropped into the src directory and I try to have different targets for them in gradle.
I also try to use the tested/versioned artefacts of e.g. apache logging instead of the unstable svn:externals version to allow reproducable builds.
For the dependencies, I try to not use the jar files in josm but instead use the official releases from e.g. maven central. This requires me to update them regularely and also adds to the differences in the build systems.

I use a different approach of generating the REVISION file that allows scanning the git tree if the repo was cloned from the git mirror. I use jgit for this to make it platform independent.

There are many more things that would need to be unified. Compiler arguments, source/target compatibility, test system properties, manifest information for the JAR file, ...

And I am using some ugly hacks in those build scripts, e.g. for working around the cyclic dependency of epsg depending on scripts depending on main which in turn depends on epsg

As you see, maintaining two build systems officially and keeping them in sync is not so easy, especially with the current, historical layout of josm. Updating the version numbers is the easiest part there and most of the time only svn:externals break, so the xml file won't save more work than updating them by hand ;-)

[stoecker]

Regarding Git/GitHub there are news:

​https://news.microsoft.com/2018/06/04/microsoft-to-acquire-github-for-7-5-billion/

[floscher]

Replying to stoecker:

Regarding Git/GitHub there are news:

​https://news.microsoft.com/2018/06/04/microsoft-to-acquire-github-for-7-5-billion/

This really only affects GitHub as the code hosting provider, not git itself.

At least I will from now on try to use GitLab more often where I used GitHub before. GitLab could be self-hosted in case you'd want to do that. The gradle-josm-plugin is already mirrored to GitLab for a while now and GitLab might become the new main home while GitHub becomes the miirror.

[anonymous]

Replying to floscher:

Replying to stoecker:

Regarding Git/GitHub there are news:

​https://news.microsoft.com/2018/06/04/microsoft-to-acquire-github-for-7-5-billion/

At least I will from now on try to use GitLab more often where I used GitHub before.

And bear in mind, that there is still:
git.openstreetmap.org

EDIT: wiktorn speaking

[Don-vip]

Gitlab is now ​completely free (not just the community edition) for open source projects.

[wiktorn]

I created separate ticket for Apache Ivy #16420

Patch is ready for review.

[anonymous]

Gradle cannot import org.openstreetmap.josm.RevisionTask. Am I missing something?

[michael2402]

The build script won't work with newer versions of gradle (at least not in the current version), since I needed some hacks to work around cyclic dependencies and around the issue, that all sources are kept in one directory.

If you want to, you can try fixing it, but I currently don't see any sense in trying to maintain the gradle build in it's current form, since we do not need three build systems for one project.

[anonymous]

The intention was behind that I wanted to build JOSM in IntelliJ Ultimate using built-in tools. But until now none of the written methods worked for me (see #17218).

@simon04 suggested Gradle as an alternative method. I don't want to use or improve it because as you mentioned, there is already two other solution.

For the build systems topic: I used Maven before, and I think it's easier to understand as a beginner (opposed to Gradle).

A bit off topic, but I don't understand why are compiled code in the source repository. Because of that, the repo is unnecessarily big, hard to understand the structure and not so easy to upgrade the dependencies. Faced this issue when tried to set up the build environment. Look at the image here ticket:17218#comment:4, how many folders are excluded.

[michael2402]

Replying to anonymous:

For the build systems topic: I used Maven before, and I think it's easier to understand as a beginner (opposed to Gradle).

Depends on what you want to do with it. For using Maven with JOSM, we would have to use a different build pipeline and complex configurations. Maven ist great for simple library-style projects that only compile Java, but we have a lot of conversion tasks that do more than that.

A bit off topic, but I don't understand why are compiled code in the source repository. Because of that, the repo is unnecessarily big, hard to understand the structure and not so easy to upgrade the dependencies. Faced this issue when tried to set up the build environment. Look at the image here ticket:17218#comment:4, how many folders are excluded.

Short answer: "It always was that way", "Works for me", "Don't touch a running system". Long answer: See protocols of endless discussions about build system / build structure / ...

[taylor.smock]

We are removing Java 8 support at the end of this year (see #17858).

Right now, I'm going to have to edit the following files:

• source:trunk/build.xml​ (ant)
• source:trunk/.classpath (eclipse)
• Local IDEA project file (since I use IDEA)

And then I have to do the same thing for every plugin that we decide to move off of Java 8. I might be able to avoid updating the build.xml file for them (since they have a build-common.xml file if we decide to increase the baseline), but I'll still have to update the eclipse file and my local IDEA project file.

I'd really like to have one file that we can use for builds on the command line and projects in IDEs (without requiring additional configuration). If it was a one-off, I don't think I'd be as worried, but I think we'll be able to upgrade the minimum Java version every 2 or 3 years after we drop Java 8 support.

The major IDEs (Eclipse, IDEA, Netbeans, VS Code) support:

• maven
• gradle
• Eclipse project files (there may be some quirks with this one -- in IDEA, I have to go through and set the test folders to be test folders, which is a PITA)

I don't think there is anything which supports eclipse project files on the command line, so we can remove that from consideration.

Of the remaining two (maven and gradle), I strongly prefer maven since gradle has been a PITA when upgrading major versions for the plugins I maintain. Both are widespread enough that distributions are more likely to have some method for handling dependencies declared in the build files; by contrast, ant (and by extension ivy) are ​not as widely used (<10% of developers). This means that distro maintainers will most likely have a harder time packaging JOSM with ant+ivy than they would with maven or gradle.

[taylor.smock]

I'm going to poke this again.

Does anyone have an objection to me adding a pom.xml file to JMapViewer for initial feedback? If we do switch to a maven/gradle, there will be some kind of transition period to avoid breaking automatic build systems.

Like I said in comment:57, I'd really like to have one source for configuration ("one place to change things").

[sebastic]

Supporting Maven is fine from a packaging POV as long as not too many plugins are used, Gradle is problematic as it's nearly impossible to package it since it switched to Kotlin.

I had to add Maven support in Osmosis to unbreak the Debian package build (​https://github.com/openstreetmap/osmosis/pull/137).

[stoecker]

From openSUSE I'd say the same. Ant is ok. Maven still works. When a software uses gradle I rather find another software.

[taylor.smock]

FML. Is this the year of the deprecations or something?

The IvyDE project is no longer maintained (ant just ​archived it). It will probably keep working in Eclipse for the foreseeable future, but it will stop working at some point.

I've read through the discussions leading up to that (start ​here).

It looks like it comes down to the current crop of ant maintainers not really knowing the ivy and the IvyDE codebase.

It also sounds like ivy might be going down the archival/deprecation path due to lack of maintainers. They are just starting with IvyDE first.

I expect that ivy will keep working "as-is", and so long as we only point at our nexus, we should be fairly safe from vulnerabilities.

It looks like there is a set of ant tasks for maven that should work to replace ivy (see ​https://maven.apache.org/resolver-ant-tasks/ ).

Hopefully this doesn't become a "critical" thing for me to do soon. I don't think I'll see something in the news about ivy being remotely exploitable anyway without something else as well (e.g. "and the repository must serve a malformed xml file").

[michael2402 -]

Based on my experience in the last years, the whole eclipse build system is used less and less. Eclipse has a pretty good maven + gradle handling and this is used instead. So I expect more deprecations in the eclipse build system / build support and future Eclipse updates might cause more issues like this.

As for Ant / Security issues: Most of them can only be exploitet with either access to the build server or access to the registry. As long as we trust all committers of code that is executed on the build server (or it uses containers to isolate the builds) and as long as we trust all people that can push artifacts to the registry, we should be fine.

[taylor.smock]

the whole eclipse build system is used less and less

Good to know.
I started out using Eclipse, but eventually moved to IDEA.
I just want to keep supporting Eclipse for the occasional patch contributor.

or it uses containers to isolate the builds

I don't think Jenkins does that. It would be nice (tm), just from the perspective of making it easy to debug why a test works a specific way in CI and doesn't locally.

we trust all people that can push artifacts to the registry

We proxy a bunch of "popular" registries on our nexus (see https://josm.openstreetmap.de/nexus ). I'm going to go with "no, we cannot trust all people that can push artifacts to the registry".

I do figure that if any significant vulnerability comes out, I figure I'll have a bit of time before we are at risk, since the nexus caches the upstream responses for the proxy repos. So I'd have to fix it prior to any dependency updates.

[michael2402 -]

Replying to taylor.smock:

the whole eclipse build system is used less and less

Good to know.
I started out using Eclipse, but eventually moved to IDEA.
I just want to keep supporting Eclipse for the occasional patch contributor.

I don't mean Eclipse - I belive that while eclipse is still lacking some of the more modern features of IntelliJ or VSCode, it is still maintained and used for a lot of projects.

What is getting less and less common is to define the build settings in the .settings/.classpath/.project files but instead use maven or gradle files and import them in Eclipse.

or it uses containers to isolate the builds

I don't think Jenkins does that. It would be nice (tm), just from the perspective of making it easy to debug why a test works a specific way in CI and doesn't locally.

The JOSM jenkins is not configured to do that. Jenkins supports a docker runner and state of the art is to define a docker container to run the build in (so for JOSM, one would use a Java+Ant container). The main advantage are reproducible builds and that for example an ANT update for JOSM won't break builds for the other projects.

From my experience, almost all pipelines use containers and only few special cases don't use them (when they need special access). Many build systems don't even support non-containerized builds.

Btw., I currently see a trend to do this even for local builds: IntelliJ already supports running the build in docker containers, which is still a bit difficult to set up, I see it in use for cross compiling or for projects that need "old" dependencies (old versions of libraries, ...).

we trust all people that can push artifacts to the registry

We proxy a bunch of "popular" registries on our nexus (see https://josm.openstreetmap.de/nexus ). I'm going to go with "no, we cannot trust all people that can push artifacts to the registry".

I only meant the ones we reference. In some way, we do trust all of them: We download those and include them in our build. So all users of JOSM need to trust them any way. And for build dependencies (like ant): If someone can replace them and wants to do bad things, you don't have to wait for a problem with ivy.

I do figure that if any significant vulnerability comes out, I figure I'll have a bit of time before we are at risk, since the nexus caches the upstream responses for the proxy repos. So I'd have to fix it prior to any dependency updates.

That depends on the problem, but yes, as long as we don't use any other external resources than the ones we already used, there should not be any problems. The past problems were with parsing malicious xml files (if someone can upload XML files for our dependencies, that person can also replace the acutal dependencies), invalid zip files or invalid path specifications.

Assuming a container build, none of those problems are "relevant" since they require either the source code or the dependency to be compromised and since a container build has no side effects (it only has the then compromised artifact as output), the problems with ivy would not have caused any additional security problems. So I can see why patching those took so long - there was just no urge for most build setups.

[taylor.smock]

Maven changes for JMapViewer

[taylor.smock]

In 36202/osm:

See #8269: Add initial Maven support to JMapViewer

This is largely 1-1 compatible with the targets of the ant build.xml file, with
some exceptions (see "Ant targets -> Maven targets" in CONTRIBUTING.md).

Please note that maven (by default) puts most generated data into the target
directory. This has been overridden for some targets.

For this commit, the following informational files were added:

• RELEASING.md: I got tired of looking up how to do a release for JMapViewer
• CONTRIBUTING.md: Just in case someone else wants to contribute to JMapViewer

[taylor.smock]

Sample POM files for JOSM core and two plugins

[taylor.smock]

Notes on attachment:8269.2.patch​:

• Adds maven files for:
  • JOSM core
    • Not a perfect 1-1 mapping from current ant build script
    • Doesn't support calling scripts individually, see ​exec-maven-plugin #430
    • Doesn't include lang3 in jar. But neither should the ant build. So this might (technically) be an improvement.
  • JOSM unittest (specifically for plugins)
    • This does make IDE integration a bit harder, since we want the plugins to be able to link to the test package, but most IDEs don't like it when a directory is in two separate modules, for some reason.
  • plugin root
    • This has the two plugins and JOSM core/unittest as modules.
  • apache-commons
  • apache-http
• Works with
  • CLI
  • IntelliJ
  • Eclipse
    • This is a "kind of". I think it would work better if I had another directory level to work with (I could probably fix this by moving the current pom.xml to nodist and renaming pom-josm.xml to pom.xml). EDIT: Eclipse does work better with the pom.xml moves.
  • NetBrains
    • See Eclipse

It is very much incomplete -- for example, I haven't gotten the plugins to merge the dependency jars into the plugin jar (it shouldn't be too difficult to fix when I get around to it again).

I've been setting up a new computer, and getting all the plugins "properly" set up was a PITA. Maybe it would be a bit better if I used Eclipse, but I don't. I did check to see if it was still possible to set up an IDE environment with Eclipse; the IvyDE plugin won't install anymore ("No repository found at [...]"), so I think the Eclipse files are effectively broken at this point.

What I would like to do:

• Replace most of the eclipse files with maven files (simply because they are more independent of an IDE, and may eventually be "good enough" to replace the ant build scripts with).

Problems:

• Duplicated information on dependencies
  • This may be alleviated if we switch to ​https://maven.apache.org/resolver-ant-tasks/
    • EDIT: Example: ​https://github.com/JOSM/MapRoulette/blob/master/build.xml#L23-L47

[taylor.smock]

Add pom.xml files for most plugins in svn, add generated changes to eclipse files (AKA stop missing nature spam)

[taylor.smock]

With attachment:8269.3.patch​, I think we can entirely remove the eclipse files (specifically .classpath and .project) from source control.

The only problematic plugin right now is opendata -- if you want to run mvn compile, you will need to comment opendata/modules out in the plugin root pom.xml.

At this point, I think I can freeze changes to the parent plugin pom, since I haven't had to make any in the past couple days as I added more pom.xml files for plugins in svn.

I would greatly appreciate feedback on the patch in the following environments:

• IDEs that are "fully" set up for JOSM development
• IDEs that are not set up for JOSM development
• Computers that are not macs (since that is what I wrote the files on)

[taylor.smock]

Add maven metadata (url, author), reduce amount of empty manifest entries

[stoecker]

NOTE: While I still think the established ant build is fine, it seems completely replacing all the build stuff by maven is probably the right way to go long term. Whether I like it or not, ant seems to be dying. Anyway I'd like to have a transition period where we have both. Everything else can vanish without transitions as far as I'm concerned.

[taylor.smock]

Works for me. I just applied my patch to a "clean" checkout of JOSM on Linux and realized that it didn't apply cleanly (largely due to CRLF issues with Eclipse files). And I was missing the jts/pom.xml file.

The current patches should produce jar files that are largely the same as those produced by the ant builds. One caveat: I'm replacing most of the lib files in the plugin directories with maven dependencies. I've tried to keep the same dependency version for these patches, but there is at least one jar that doesn't have an equivalent version in maven (newer versions have a different API).

In any case, ant+ivy does seem like a dead end and is a PITA for IDEs as well. We worked around the problem with IDEs by having eclipse project files, but that is two sources of configuration, and (as I found out), it was a PITA to get going with Eclipse (the project natures kept wanting to be installed, and kept popping up windows keeping them from being installed).

As I do dependency updates, I'll probably also move the current ant builds to maven-resolver-ant-tasks from ivy.

[gaben]

@taylor, could you please upload an svn patch? The 8269.4.patch​ seems to be a git patch.

[stoecker]

Replying to gaben:

@taylor, could you please upload an svn patch? The 8269.4.patch​ seems to be a git patch.

There's no difference. Try patch -p 1 <8269.4.patch​, that strips the first directory level.

[taylor.smock]

@gaben: I don't know how you apply patches, but I typically do something like patch -p0 < file.patch. For git patches, it is patch -p1 < file.patch.

As noted, there are some issues with 8269.4.patch that I discovered when I went and tried applying it on a Linux box (the patch was written on a mac box).

Of specific note, there were some issues with line endings. And there were some files that shouldn't have been included (specifically pom.xml files for plugins that aren't yet in JOSM github). So you'll get a lot of errors when you apply it (mostly related to the line endings).

EDIT: What I have locally is probably a bit better than 8269.4.patch since I just deleted all the other IDE specific files. But I also deleted a lot of jar files and am trying to figure out what breaks with that.

[taylor.smock]

Just the pom.xml files

[taylor.smock]

I'll go ahead and apply what I have tomorrow or Friday, absent objections.

[taylor.smock]

In 19097/josm:

Fix #8269: Add initial maven pom files

This additionally deletes most IDE specific configuration, in favor of Maven
configuration.

We did have to update scripts/BuildProjectionDefinitions.java so that it would
create an empty EPSG file for the next run in the pom.xml file.

For core, we needed two additional pom.xml files:

• A parent pom with most dependency information (for inheritance)
  • This will make it easier to keep core and the apache-commons plugin in sync
• A unit test pom (for plugin tests)
  • Some IDEs (netbeans specifically) don't like this.

What this does not do:

• Convert from ivy to maven for dependency management in core. This should happen eventually with maven-resolver-ant-tasks (probably with the next dependency update).
• Deprecate the current ant scripts.

[GerdP]

With this change Eclipse misses its project file. What am I supposed to do?

[taylor.smock]

I don't know if there is a better way, but

1. Delete the modules from eclipse (don't delete files from disk!)
2. File -> Import -> Existing Maven Projects

I didn't think anyone was using the eclipse project files since when I tried setting up an eclipse environment, I got a bunch of nature request popups that literally prevented me from doing anything without a lot of work.

EDIT: You might be able to just do step (2), but I'm not certain on that.

[taylor.smock]

In 36282/osm:

Fix #8269: Add initial maven pom files

This additionally deletes most IDE specific configuration, in favor of Maven
configuration.

What this does not do:

• Convert from ivy to maven for dependency management. This should happen eventually with maven-resolver-ant-tasks (probably with the next dependency update).
• Deprecate the current ant scripts.

[GerdP]

OK, I guess I just wait until the wiki https://josm.openstreetmap.de/wiki/DevelopersGuide/Compiling was updated and then start from scratch.

[taylor.smock]

I've updated it slightly with some basic instructions for Eclipse. I need to check and see if the remaining bits of the Eclipse setup (old) are still necessary (w.r.t. epsg-touch and mappaint; I don't think so).

[GerdP]

I followed the instructions. Eclipse is now "Updating Maven projects" for quite a while. It counts up to +90% and starts again. Is there a way to find out when it will finish?

[GerdP]

I closed all projects and that helped. Seems I can use Eclipse again.

[taylor.smock]

Replying to GerdP:

I followed the instructions. Eclipse is now "Updating Maven projects" for quite a while. It counts up to +90% and starts again. Is there a way to find out when it will finish?

On my mac, it took something like 10 minutes for it to finish updating maven projects (i9 2.4 GHz).

I closed all projects and that helped. Seems I can use Eclipse again.

Good to hear.

[GerdP]

BTW: Since this change any edit in a *.java file seems to trigger a lot more "building ..." actions then before or maybe "building with maven" is just much slower compared to the method that was used before?

[taylor.smock]

Eclipse has its own internal build mechanism which isn't (apparently) used with maven projects.

It does seem to be building more than I expected. Maybe eclipse is still trying to compile for every save that is made?

I was testing by adding a new constructor to BBox (ILatLon instead of LatLon), and it took a little while to build. It took ~1 minute on my old (i5-4690) machine. To be semi-fair, I was making a change to BBox which does affect a lot of classes.