Move JOSM plugins without maintainer back to JOSM-Plugins-SVN (GitHub SVN support ends)

[stoecker]

As the SVN-Support of GitHub ends

​https://github.blog/2023-01-20-sunsetting-subversion-support/

either the external plugins in JOSM group are moved back to JOSM-Plugins-SVN or they are completely on their own for I18N and release handling.

WIP repo: ​https://github.com/tsmock/josm-osmsvn-svn2git

[taylor.smock]

they are completely on their own for I18N and release handling

I can probably write an automatic i18n update script for them, and I've got release handling done in a scripted manner (just incremental revisioning though, not semver).

In any case, I'd rather not move from git to svn -- I use IDEA in a flatpak for my home dev environment, and it doesn't have access to svn, which makes figuring out why some piece of code was written much harder. I can still commit via the CLI.

I think I'll look into creating a monorepo at github.com/JOSM/josm-all.

[stoecker]

they are completely on their own for I18N and release handling

I can probably write an automatic i18n update script for them, and I've got release handling done in a scripted manner (just incremental revisioning though, not semver).

Well. That's policy. Either JOSM hosted or on their own. I made an exception for the JOSM plugins in GitHub JOSM group for various reason which mainly never became true. If this vanishs then it's back to the original. Any additional method maybe easier for some users, but it greatly increases the maintenance costs on my side and that's simply not anything I want. If it's too much work one day I simply may say it's not worth the effort and that's probably not so good a situation.

Major reason was this:

• When moving to GitHub the authors will care for their plugins as they all use Git and not SVN. Looking at the timeline the GitHub plugins get abandoned same as SVN only they make much more work.

[taylor.smock]

but it greatly increases the maintenance costs on my side and that's simply not anything I want

That is totally fair. You just want to have one VCS system to maintain/worry about.

I'll spend some time looking into migrating from svn to git, specifically with the following in mind:

• Map svn users to git users; I may want to reach out to old contributors to figure out what they want me to use.
• Map patch by <foo> to be authored by <foo> but committed by the git user mapped from the svn committer.

I do know that a good chunk of patches from third-party contributors use git as the VCS (patch -p1 instead of patch -p0 due to a/ and b/ prefixes).

Looking at the timeline the GitHub plugins get abandoned same as SVN only they make much more work.

Speaking for myself, the major problem has been ensuring I've got the right repo URL set up -- I commit i18n updates to both svn and git at the same time. And then going through and updating the release version (which I'm slowing moving away from, as I add the autorelease GitHub action).

Additional notes:

• ​Trac natively supports git since 1.0
• ​GCC had success moving from svn to git using ​reposurgeon, so I'll probably try that first.

[stoecker]

Issue with Git is that most people with "git" don't mean "git", but rather "GitHub". Setting up a git on the server and hosting the code there would be easy. But nearly nobody who requested that would actually be happy with that. What they want is the Fork and PullRequest and whatever options of GitHub which actually make working with Git "nice".

And that's where all the procedures fail.

P.S. As far as I know if you prefer working with git rather than SVN, then simply using git-svn to checkout the SVN is possible. Then you can work with it like it would be a git repo. You (and your tools) don't see the difference afterwards.

So the abandoned plugins go back in the JOSM-Plugins-SVN (probably loosing the commit history [or with little more effort only the commit date]) and you access it simply as one git.

[taylor.smock]

Issue with Git is that most people with "git" don't mean "git", but rather "GitHub".

Fair. I mean 'git'.

And that's where all the procedures fail.

I think I need some additional information here. I assume that you are talking about a migration from svn to git + GitHub/GitLab/Gitea/whatever.

What they want is the Fork and PullRequest and whatever options of GitHub

They aren't just features of GitHub; most platforms that support git as the primary VCS have those (Gitea, GitLab, etc.). I could do without those features personally, but they are "nice-to-have". With that said, I'm open to investigating self-hosting one of the OSS options with the intent of making it one-one as possible with Trac.

then simply using git-svn to checkout the SVN is possible

This requires svn. I've used it before. And still use it to update ​https://github.com/JOSM/josm-plugins .

Beyond that, what can I do to make a transition to git possible? If it is something that will never happen, just let me know. If it is something that could happen, given <x>, <y>, and <z>, at least tell me what <x>, <y>, and <z> are.

[stoecker]

I assume that you are talking about a migration from svn to git + GitHub/GitLab/Gitea/whatever.

Yes. Maybe I'm old fashioned, but I'm maintaining JOSM for over a decade now and JOSM's is still there, while I've seen so many other fancy software, systems and platforms come and go that my approach can't be so wrong.

If it is something that will never happen, just let me know.

I'll never say "never". ATM I tend to "will not happen". As said Git alone doesn't change much and Gitea, GitLab, etc. are no option. Vincent spent a lot of time one this and got no real result and I'm for sure not willing to spend the time needed for creating or maintaining this.

I'm still open to migrate JOSM-Plugins-SVN to Git (and only git) hosted on JOSM server, but actually I see no big advantage over SVN in this case, as we don't really have concurrency.

But if you can convince me that this would open a wonderful new future and we can reimport all the JOSM plugins into it, maybe that's a way to go. The additional workload to setup a plain git repo to replace the JOSM-OSM-SVN should be minimal. Probably I can even setup git based on git-svn on top of the existing SVN thus offering both ways?

[taylor.smock]

I've pushed a WIP repo to ​https://github.com/tsmock/josm-osmsvn-svn2git. I've been writing the conversion script with an eye towards applying it to JOSM core as well. The big thing is mapping users -> name + email, and patch by to the git Authored By attribute.

Notes:

• I need to get a map of svn user -> Name + email. I'll be sending an email out on josm-dev asking for that information. Important note: the email addresses will be part of the public git commit history, which is why I'm not asking for a map of svn user -> email from you.
  • I've used emails that were already in the commit log or were otherwise easily available on github
• I've removed the dist directory for now -- it adds 5.6GB to the repo size.
  • If we continue using the dist directory for releases, it should use git lfs -- does the server we're using support lfs?
• The monorepo will be split up once we've finalized:
  • The author map
  • Which directories will be split out
• svn:externals were not converted to git submodules; this might be a manual step. I might be able to get reposurgeon to convert the github externals for me. The core externals are probably going to have to be symbolic links.
• Attribution samples
  • Co-authored-by: ​473116
  • Authored by: ​fedb2a

Git repos to split out of the monorepo:

• viewer/jmapviewer -> jmapviewer
• editors/josm -> josm-root (for submodules)
• editors/josm/dist -> dist (git config -f .gitmodules submodule.dist.shallow true to avoid cloning the whole thing if our server doesn't support git lfs)
• editors/josm/i18n -> i18n (maybe)
• editors/josm/mvt -> mvt (maybe)
• editors/josm/oldplugins/(.*) -> $1 (directories only, these would be "archived" repositories if we were using something like GitHub)
• editors/josm/plugins/(.*) -> $1 (directories only)

[taylor.smock]

reposurgeon scripts

[marcello@…]

Great move!

If you are going to use "something like GitHub" you can use github assets for the /dist directory.

You can enumerate your assets through the github API and it is easy to write a task for JOSM that downloads plugins from assets.

[stoecker]

• I need to get a map of svn user -> Name + email. I'll be sending an email out on josm-dev asking for that information. Important note: the email addresses will be part of the public git commit history, which is why I'm not asking for a map of svn user -> email from you.
  • I've used emails that were already in the commit log or were otherwise easily available on github

Wouldn't help to ask, as I have no such list.

[taylor.smock]

Replying to marcello@…:

Great move!

Some people might disagree...

If you are going to use "something like GitHub" you can use github assets for the /dist directory.

I don't think we are going to use "something like GitHub" anytime soon. Maybe if Trac loses all maintenance support or there are significant vulnerabilities found that aren't fixed in a reasonable timeframe.
With that said, most of the plugins I've been updating on GitHub have had their download URL changed to https://github.com/<group>/<repo>/releases/latest/download/<jar>. I just have to manually set the old version links in the manifest -- the svn dist directory has that done automatically.

You can enumerate your assets through the github API and it is easy to write a task for JOSM that downloads plugins from assets.

At this point, I wouldn't do this. If there is a third-party forge that has the same api, I might be more inclined to do that. I don't feel like writing an integration for:

• GitHub
• GitLab
• Gitea (this might have a similar API as GitHub -- they at least claim to be compatible with GitHub actions anyway)
• <Some other random git forge>

Replying to stoecker:

Wouldn't help to ask, as I have no such list.

I would have figured that it would be something that Trac had in its database. As noted in the email I sent out on JOSM-dev yesterday, firefishy has a list he used for the OSM svn migration. The big thing about using email -> username mappings from something like Trac is that the emails might not have been intended to be public. And I don't want to accidentally disclose someone's email.

I'll be going through patches that we've applied in the past to see if there is git committer information in them as well. I might want to script that though.

[taylor.smock]

Ticket retargeted after milestone closed

[GerdP]

Is #23528 a duplicate of this ticket? If yes, what does it mean for me? Is there a work around that doesn't involve the use of a git client?

[stoecker]

Ticket #23528 has been marked as a duplicate of this ticket.

[stoecker]

We should finally setup the Git for plugins repo.

@Taylor: What's the state of this? Can we go on? Actually wanted to ask some time because of the error mails I get, but there's such much else to do...

[taylor.smock]

Pretty much done. I just need to do the following:

• Split the plugins into individual repositories
• Perform a run where I reformat all the files to the current standard (this is literally just to make it easier to look back at history)
• Make certain the code I wrote for converting svn:externals to .gitmodules is working properly

Notes on the conversion:

• I'm pretty much dropping the mega deletion commit that Don-vip reverted
• There are some series of commits that were munged together (where I could tell that the user was effectively using svn to save their workstate; the commit messages were effectively all the same, e.g. "Modify <foo>" or "save")
• I haven't gotten any more takers on the svn username -> git attribution. I might be able to track down some more if someone looks through the Trac user database.
• Any plugin that used svn:externals aren't going to get their own .gitmodule right now, but that is mostly for historical purposes. I think that information will end up being tracked in the master repo .gitmodule file.

I'll try to get that done in the next few days.

[taylor.smock]

@stoecker: Have we decided what to do for the /dist directory?

I'm kind of thinking of reusing our maven repo for storing the plugin jars.

[GerdP]

This all sounds as if I have to get git running on my windows systems? The last time when I tried that it was a desaster because git (or tortoiseGit) messed up the sources. IIRC the different line endings on the different OS were hard to handle with the git tools.
That was some years in the past but I fear that this is still a problem. Maybe because my preferred workflow with patches simply cannot be used with git or I used the wrong commands.

[taylor.smock]

IIRC the different line endings on the different OS

git should handle line endings pretty well now. There is a setting as well (see ​https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings ).
With that said, one of the conversion steps should be converting the files to unix line endings. I can add a .gitattributes to all the repos if you think this will be a major issue going forward.

Maybe because my preferred workflow with patches simply cannot be used with git or I used the wrong commands.

Probably the wrong commands. I usually use curl $(patch_url) | patch -p$(1 or 0, depending upon whether or not the patch is from git or svn).
There is also an apply patch option for git which will keep contributor information. Assuming that the patch was created using git format-patch, git am is the right command (just pipe the patch into it, either via curl $(patch_url) | git am or git am < $(patch_file)).

In short, try checking out the linked repo in the original comment. There are some issues (since I haven't mirrored all of the GH JOSM plugin repos into my own account; I'll do that now, but it might take a bit).

[GerdP]

With that said, one of the conversion steps should be converting the files to unix line endings

I think that is one of the reasons for the problems that I had. If you or git convert files old patches may no longer work without additional work.

Probably the wrong commands. I usually use ...

This requires to work with a unixoid shell. My experience with those (esp. on Windows) are rather bad and probably another reason for the messed up sources as it is always unclear whether the called programs are Windows tools or unix tools.

Anyway, I've installed latest TortoiseGit and I'll see if I am able to work with a mixture of svn and git when I am forced to.

[taylor.smock]

old patches may no longer work without additional work

Once this is done, I'll go through and make certain that old patches apply cleanly. That will be a good excuse to go through them in order to merge/reject.

I'll see if I am able to work with a mixture of svn and git

I think the primary problem is going to be JOSM core in svn. I'll check and see how "easy" it is to use git-svn with the JOSM git repo. Note on the current JOSM git repo: history doesn't go back to r1. I've got a version which does, but I don't think it is compatible with git-svn. I'll check.

[GerdP]

My understanding is this: A checkout from svn and the result of a git clone should produce identical files (content and meta data like last update timestamp etc). I can live with the additional .gitignore file but different timestamps or different line endings are problematic for my work flow. My first attempt with ​git clone ​https://github.com/tsmock/josm-osmsvn-svn2git shows various differences in the files in \oldplugins reg. line endings.
I think I've configured svn to create the file with a time stamp of the commit. When I use svn revert it reverts a file also to that timestamp.
I have to learn how these basic goals can be achived with git.

[taylor.smock]

shows various differences in the files in \oldplugins reg. line endings

I didn't see any when I checked it out. But I'm on linux. This might be a CRLF issue.

I think I've configured svn to create the file with a time stamp of the commit. When I use svn revert it reverts a file also to that timestamp.

What is the use case? Depending upon what you are trying to do, the ​git hooks is what you want.

[taylor.smock]

I should probably add on:

A checkout from svn and the result of a git clone should produce identical files (content and meta data like last update timestamp etc)

The tool I am using tries to be as faithful to the original repo as possible given the implemented features and assuming the user doesn't want to clean up the history.
I'm trying to clean up the history a bit using the tool. Of note:

• Convert line endings from dos to unix
• Patch commits have the right author
• Some commits have been modified
  • r30196/osm and r30197/osm have been merged
  • r29884/osm..r29904/osm have been merged
  • r30279/osm..r30297/osm have been merged

[GerdP]

This might be a CRLF issue.

Sure, it always is ;)
Exampe: files in oldplugins\download_along\src\org\openstreetmap\josm\plugin\download_along

 Verzeichnis von d:\git_repos\josm-osmsvn-svn2git\applications\editors\josm\oldplugins\download_along\src\org\openstreetmap\josm\plugin\download_along

01.03.2024  13:54    <DIR>          .
01.03.2024  13:54    <DIR>          ..
01.03.2024  13:54               596 DownloadAlong.java
01.03.2024  13:54             7.220 DownloadAlongWayAction.java
               2 Datei(en),          7.816 Bytes
               2 Verzeichnis(se), 992.597.741.568 Bytes frei

 Verzeichnis von C:\josm\oldplugins\download_along\src\org\openstreetmap\josm\plugin\download_along

10.04.2019  07:49    <DIR>          .
10.04.2019  07:49    <DIR>          ..
17.01.2019  12:10               596 DownloadAlong.java
28.01.2019  10:40             7.393 DownloadAlongWayAction.java
               2 Datei(en),          7.989 Bytes
               2 Verzeichnis(se), 150.293.094.400 Bytes frei

In the checkout from svn the file DownloadAlongWayAction.java has crlf endings, the other has lf. From git both files are with lf and date is today instead of commit timestamp.
The mixture of line endings is quite common in the plugin sources, at least on my machine.
I've configured git to keep the line endings as they are (AutoCrLf false), I assume that is the default, no idea how to find out.

[taylor.smock]

The mixture of line endings is quite common in the plugin sources, at least on my machine.

One of the things I'm trying right now is running a formatter on the java files to bring everything in conformance with the josm defaults. So it is easier to skip commits that were "just" formatting commits when digging through history.

[GerdP]

What is the use case? Depending upon what you are trying to do, the ​git hooks is what you want.

Did not find anything that sounds useful in the hooks description, but I think I was able to configure git to do this in the past (which might be 15 years ago). Maybe it is or was an undocumented configuration option?
My favorite tool is the Total Commander (TC, an NC clone) and in my eyes it is extremely useful to have the files sorted by age so that the newest are at the top of the listing. That's what I am used to since +30 years, so it's hard to accept that this doesn't work when working on directories from git.

[taylor.smock]

files sorted by age

Last modified? First created?

It ​looks like someone wrote a tool for last modified. You could probably put that in a hook.

[GerdP]

OK, there are probably reasons why git doesn't provide that functionality, so I'll rather keep it simple. If this migration happens in my life time and I still want to fix problems in JOSM or its plugins I'll find out how to work around the problems with git as well ;)

[taylor.smock]

The primary reason (from what I can tell) is that some tools depend upon the last modified time can have issues when switching between branches.
I think you would be fine using the linked tool for setting last modified time.

[taylor.smock]

Ticket retargeted after milestone closed

[taylor.smock]

Ticket retargeted after milestone closed

[taylor.smock]

Ticket retargeted after milestone closed

[taylor.smock]

Ticket retargeted after milestone closed