Migrate GitHub mirroring script to josm.openstreetmap.de server

[simon04]

The script has been running on my PC at ​since 2015-04-07. I wouldn't have expected it to be still relevant and in production 5.5 years later.

The mirroring process is mission critical now, #19937.

I don't want to run and maintain this mirroring script any more. It does not belong to a private PC of a contributor. I'm happy to share a tarball of everything I have.

[Don-vip]

@Simon can you please share the tarball with Dirk and I? I have no idea what the mirroring script consists of.

[simon04]

I've sent you both an email.

[Stereo]

I could also have a look at running it as a github action if you want?

[stoecker]

Milestone renamed

[Stereo]

The scripts haven't been running for a week.

Having the code on github is necessary to create macOS releases.

[stoecker]

Ah, he wont let me off the hook... Tsss.

[Stereo]

@stoecker is it only you who can get this to run on the josm server?

[stoecker]

Replying to Stereo:

@stoecker is it only you who can get this to run on the josm server?

Also Vincent :-)

[stoecker]

I'll see what I can do this week.

[Stereo]

Something like ​https://trac.edgewall.org/browser/trunk/contrib/trac-svn-hook to get instant pushes?

[Stereo]

@stoecker can I help?

[stoecker]

Nah, I had a look yesterday. It's easy. It's only my typical problem: Little motivation after a long day in the office at the computer :-) But I see release is coming... I will get it done (at least I believe so :-).

[Stereo]

That's what I'm offering to help with :).

[stoecker]

Whah, it only LOOKED easy, it's not. From what I got from Simon I extracted following stuff as a perl script (was bash before, but as it only calls commands it makes no difference), but it does not work. Also it's missing the initial setup.

I'd like to have a working commandchain

• for the case the local git is established (i.e. the directory exists)
• for the case nothing is there (i.e. the directory does not exist)

Maybe you can help here? Actually I don't want to fiddle with git details until I know what's wrong here. My issue with git is that the syntax and commands really gets unintuitive as soon as you go a bit deeper in what git really differentiates from SVN. The commands look very similar to what I tried 2 years ago and failed with :-)

#!/usr/bin/perl -w

use strict;

my $home = "/home/josm/auto";

sub mirror
{
  for my $name ("josm", "josm-plugins")
  {
    chdir "$home/josm-mirrors/$name";

    # Commit externals changes, if any
    system("git","config","user.name","JOSM GitHub mirror");
    system("git","config","user.email","mirror\@josm.openstreetmap.de");
    system("git","config","user.password","...");
    #system("git","config","credential.helpe","store");

    # Pull changes from Subversion
    system("git","checkout","master");
    system("git","svn","fetch");
    system("git","svn","rebase");

    # Push the mirror to GitHub
    system("git","remote","add","mirror","git\@github.com-josm:openstreetmap/$name.git");

    # Push to our mirrors
    system("git","push","mirror","master");
  }
}

mirror();

[Stereo]

Where is it failing? And... why are you rewriting it in perl? :)

I'm bootstrapping a svn copy and perl etc. but it might take me a while. Would it please be possible to get access to the server?

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN8OQNxjZtd5aamPItqlEqAjbw7dOfSyXi2eMI5BFFfr stereo@StrangerHereMyself

[Stereo]

Ah I see, it's failing because it's not a git repository? It needs to be both a git and a svn repository?

[stoecker]

Replying to Stereo:

Where is it failing?

Sent you something by mail

And... why are you rewriting it in perl? :)

So that I have only one common infrastructure and not a different method for each task.

[Stereo]

Ok, I give up on trying to make git-svn work on macOS. It's a nightmare to get to compile, and then it crashes with messages Google has never seen:

dyld: lazy symbol binding failed: Symbol not found: _svn_ra_make_callbacks
  Referenced from: /usr/local/Cellar/perl/5.32.1_1/lib/perl5/site_perl/5.32.1/darwin-thread-multi-2level/auto/SVN/_Ra/_Ra.bundle
  Expected in: flat namespace

Let's hope debian works better.

[Stereo]

​https://github.com/openstreetmap/openstreetmap-mirror/blob/master/svn-mirror.sh is probably the init script we want

[Stereo]

Even after re-initialising a repo that way, it doesn't want to push to master:

stereo@debian:~/josm/josm$ git push mirror master
To github.com:grischard/josm.git
 ! [rejected]              master -> master (non-fast-forward)
error: failed to push some refs to 'git@github.com:grischard/josm.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

I'm not sure I understand why, and I'm not sure how to fix it. Running git pull complains about "fatal: refusing to merge unrelated histories".

Indeed, it looks like the commit hash my own mirror has come up with aren't the same hashes github currently has. There must be some kind of state file somewhere.

[Stereo]

​https://github.com/grischard/josm progress!

The .git/svn/refs/remotes/trunk from simon04 seems to contain the whole state that maps svn commits to git commits.

There's index which is a binary file, and unhandled.log which contains I'm not sure what. I'm also not sure that this is the only state information that's required.

I'd suggest cutting our losses, keeping the thing running as it is for now, and preciously keeping a backup.

[stoecker]

Seems to work now.

Still I'd like a method to work without a necessary backup. What if the server fails after a push, then the backup will not have the current state.

Can't the SVN-git files be added to the git repo in the process?

[Stereo]

Cool, thank you! And the pre-releases are building at ​https://github.com/openstreetmap/josm/releases again.

Adding commits on top of a repo is a recipe for a mess - I should know, that's how I maintained the macOS patches for a while :). I could maybe try to come up with a solution to push the metadata to a different repo?

Can you maybe paste your latest version of your script here so I can build from that? Different ticket so we can close this and do a release?

[stoecker]

I switched the mirror URL in the repo to the one below and added SSH key to the account, then it worked (password based access didn't work).

sub mirror
{
  for my $name ("josm")
  {
    my $logfile = "/home/josm/trac/log/git_$name.log";
    chdir "/home/josm/josm-mirrors/$name";

    # Commit externals changes, if any
    system("git config user.name \"JOSM GitHub mirror\" >$logfile 2>&1");
    system("git config user.email mirror\@josm.openstreetmap.de >>$logfile 2>&1");

    # Pull changes from Subversion
    system("git checkout master >>$logfile 2>&1");
    system("git svn fetch >>$logfile 2>&1");
    system("git svn rebase >>$logfile 2>&1");

    # Push the mirror to GitHub
    #system("git remote add mirror git\@github.com:openstreetmap/$name.git >>$logfile 2>&1");

    # Push to our mirrors
    system("git push mirror master >>$logfile 2>&1");
  }
  chdir $home;
}

[Don-vip]

I guess this is fixed?

[Stereo]

Yeah. I'll open a followup ticket for the backup script.