GitHub out of date

[simon04]

JOSM's ​GitHub mirror is out of date and hasn't been updated since Sept, 19, 2011.

[olejorgenb]

Any word on this?

[stoecker]

This is no official part of JOSM. The responsible persons need to be contacted directly (whoever they are).

[simon04]

I filed an issue in GitHub: ​https://github.com/openstreetmap/josm/issues/1

[bastiK]

It is run by Ævar Arnfjörð Bjarmason, I also contacted him 2 days ago.

[akks]

The mirror is outdated again.

If we had fast-updating github mirror, we could also use Travis CI (automatic building and testing after each commit) and detect broken commits.

[simon04]

This change would hopefully fix the mirror again: ​https://github.com/openstreetmap/openstreetmap-mirror/pull/2

The mirror script is here ​https://github.com/openstreetmap/openstreetmap-mirror/blob/master/josm-mirror.sh

[simon04]

Ticket #9483 has been marked as a duplicate of this ticket.

[simon04]

Brought GitHub mirror back to life. Now, it runs on my personal VPS.

This git-svn thing isn't always easy, especially to reattach a Git clone to a SVN repository. Approximately, this is what I did (with the help of ​http://stackoverflow.com/a/13105418/205629):

git clone https://github.com/openstreetmap/josm.git
git fetch --all
git checkout master
git config svn-remote.svn.url http://josm.openstreetmap.de/svn/trunk
git config svn-remote.svn.fetch :refs/remotes/trunk
git update-ref refs/remotes/trunk c96225bf2118bd2bd4692b155e5f4575172f828c # corresponding to the latest commit of branch master
git svn fetch
git svn rebase

Now, I run the syncing in a cronjob

*/5 * * * * bash /home/simon/mirrors/openstreetmap-mirror/josm-mirror.sh >/dev/null 2>/dev/null

where josm-mirror.sh is from ​https://github.com/openstreetmap/openstreetmap-mirror with this modifications

josm-mirror.sh

@@ -1 +1 @@
 #!/bin/bash -e
 
+cd ~/mirrors/josm/
+
 # Changes already pulled, merge them to the mirror branch
 git branch mirror || :
 git checkout mirror
@@ -18 +20 @@
 git add .
 git commit -m"josm-mirror: bumped externals" || :
 
-# Evil revision hack
-perl -pi -e 's[<arg value="."/>][<arg value="http://josm.openstreetmap.de/svn/trunk"/>]g' build.xml
-git commit -m"josm-mirror: evil build.xml revision hack" build.xml || :
-
 # Push the mirror to GitHub
-git remote add mirror git@github.com:openstreetmap/josm.git || :
+git remote add mirror git@github.com-josm:openstreetmap/josm.git || :
 
 # Push to our mirrors
 git push -v mirror master

[bastiK]

This is great, thanks Simon!

[Don-vip]

Thanks!

[simon04]

The update wasn't running since I forgot the git svn commands … Here's the current diff against the openstreetmap-mirror repo:

josm-mirror.sh

@@ -1 +1 @@
 #!/bin/bash -e
 
+date -Is
+
+cd /home/simon/mirrors/josm/
+
+# Pull changes from Subversion
+echo $ git checkout master
+git checkout master
+echo $ git svn fetch
+git svn fetch
+echo $ git svn rebase
+git svn rebase
+
 # Changes already pulled, merge them to the mirror branch
+echo $ git branch mirror
 git branch mirror || :
+echo $ git checkout mirror
 git checkout mirror
-git merge master
+echo '$ git merge master -m"Merge branch 'master' into mirror"'
+git merge master -m"Merge branch 'master' into mirror"
 
 # Just do a plain copy of the externals into this repository.
+echo $ svn export ....
 svn export --force http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/bzip2                    src/org/apache/tools/bzip2   >/dev/null
 svn export --force http://svn.openstreetmap.org/applications/viewer/jmapviewer/src/org/openstreetmap/gui             src/org/openstreetmap/gui    >/dev/null
 svn export --force http://svn.openstreetmap.org/applications/share/map-icons/classic.small                           images/styles/standard       >/dev/null
@@ -15 +31 @@
 git config user.name "JOSM GitHub mirror"
 git config user.email "openstreetmap@v.nix.is"
 
+echo $ git add .
 git add .
+echo '$ git commit -m"josm-mirror: bumped externals"'
 git commit -m"josm-mirror: bumped externals" || :
 
-# Evil revision hack
-perl -pi -e 's[<arg value="."/>][<arg value="http://josm.openstreetmap.de/svn/trunk"/>]g' build.xml
-git commit -m"josm-mirror: evil build.xml revision hack" build.xml || :
-
 # Push the mirror to GitHub
-git remote add mirror git@github.com:openstreetmap/josm.git || :
+echo $ git remote add mirror git@github.com-josm:openstreetmap/josm.git
+git remote add mirror git@github.com-josm:openstreetmap/josm.git || :
 
 # Push to our mirrors
+echo $ git push -v mirror master
 git push -v mirror master
+echo $ git push -v mirror mirror
 git push -v mirror mirror
+
+echo Done ...
+echo

[Don-vip]

The mirror is broken again :(

[simon04]

From: wiktorn
Date: Fri, Oct 28, 2016 at 8:32 PM
Subject: Github mirror not updated
To: josm-devel <josm-dev@openstreetmap.org>


Hi,

It looks like GitHub mirror of OSM is not updated any more. Last
update was on 21st of October.

What needs to be done to bring this service up?

Or shall I just setup my own sync service with my JOSM GitHub mirror?


Cheers,

Wiktor

---

From: simon04
Date: Sat, Oct 29, 2016 at 12:40 AM
Subject: Re: Github mirror not updated
To: wiktorn
Cc: josm-devel <josm-dev@openstreetmap.org>


Hi,

I run the mirror. The script is located here:
https://github.com/simon04/josm-mirror and see
https://josm.openstreetmap.de/ticket/6887 for earlier problems.

The cause for the current broken-ness is:
$ git svn rebase
Checksum mismatch: src/org/openstreetmap/josm/data/coor/EastNorth.java
77fca040915bef6c6fd77e43c35d952c684b9d3d
expected: e3c9169bad42e3154887b60a20bbea8b
     got: da2689e0e3989a6c2bc5b5449ad570eb

I'll try to resolve this asap (tomorrow evening?).

Simon

[simon04]

I revived the mirror using the commands from comment:10.

@team: In case you want your JOSM commits liked to your GitHub account, follow ​https://help.github.com/articles/why-are-my-commits-linked-to-the-wrong-user/#commits-are-not-linked-to-any-user

[wiktorn]

Replying to simon04:

I revived the mirror using the commands from comment:10.

Great, many thanks.

@team: In case you want your JOSM commits liked to your GitHub account, follow ​https://help.github.com/articles/why-are-my-commits-linked-to-the-wrong-user/#commits-are-not-linked-to-any-user

Just to ease this for everybody. Email to add to GitHub profile is in format:
[username]@0c6e7542-c601-0410-84e7-c038aed88b3b

(for this mirror)

[wiktorn]

Once again synchronization stopped working :-/

[stoecker]

Maybe we should move the mirror script to JOSM server?

[Don-vip]

Simon, is there something we can do to help?

[simon04]

Sorry, for the long delay. I haven't had time to read through all JOSM Trac mails. Feel free to mail me personally.

The mirror should be up again. I wouldn't mind getting rid of the mirroring process. IMO it will always remain hacky unless we get rid of all SVN externals (and obtain build dependencies using Apache Ivy or similar) or switch to Git completely.

[wiktorn]

It looks like some externals are missing in Git mirror:
e.g:
​https://github.com/openstreetmap/josm/tree/mirror/src/org/apache/commons/compress/

Is missing utils. Maybe it's worth to review all externals.

Replying to simon04:

switch to Git completely.

+1 for switching to Git from me :-)

[michael2402]

Last github sync was on April 11.

And by the way, +1 for getting rid of the build dependencies (which would probably start a new which-depencency-management-system-is-best war)

[simon04]

I brought the mirror back to life, again. Did not investigate what caused the problem. I strongly hope that we can get rid of this hacky mirroring process, eventually.

[simon04]

Via mail from bafonins:

I prefer using git instead of svn and till this moment it worked fine, but after the latest merge from the main repository to my fork I am not able to build the project. This is what I get :

compile:
    [javac] Compiling 378 source files to /Users/bafonins/Projects/josm2/build
    [javac] Projects/josm2/src/org/apache/commons/compress/compressors/lzw/LZWInputStream.java:25: error: cannot find symbol
    [javac] import org.apache.commons.compress.MemoryLimitException;
    [javac]                                   ^
    [javac]   symbol:   class MemoryLimitException
    [javac]   location: package org.apache.commons.compress
    [javac] Projects/josm2/src/org/apache/commons/compress/compressors/lzw/LZWInputStream.java:124: error: cannot find symbol
    [javac]             throws MemoryLimitException {
    [javac]                    ^
    [javac]   symbol:   class MemoryLimitException
    [javac]   location: class LZWInputStream
    [javac] Projects/josm2/src/org/apache/commons/compress/compressors/lzw/LZWInputStream.java:133: error: cannot find symbol
    [javac]                 throw new MemoryLimitException(memoryUsageInKb, memoryLimitInKb);
    [javac]                           ^
    [javac]   symbol:   class MemoryLimitException
    [javac]   location: class LZWInputStream
    [javac] Note: Projects/josm2/src/oauth/signpost/AbstractOAuthProvider.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 3 errors

[simon04]

The SVN externals have changed. Updated the mirroring code ​https://github.com/simon04/josm-mirror/commit/35c2c2e5b7abc0f0463c836e0489983bfbab4a36. Starting with this commit, ​https://github.com/openstreetmap/josm/commit/2f9a23ea744bce42c33ee52aaf82135732d0950f, the build should be working again.

[Don-vip]

Ah sorry, didn't realize it would break the git mirror. I will make a PR if I have to change externals again.

[Don-vip]

​https://github.com/openstreetmap/josm-plugins is broken (no update since november). Can you please look into it? Or tell me how to solve it :)

[floscher]

​https://github.com/openstreetmap/josm is also no longer updated for 8 days now.

[stoecker]

Can you make the mirror script available for hostong on JOSM server or whereever necessary?

[stoecker]

If we can't get the mirror active again it should be disabled.

[wiktorn]

Replying to stoecker:

Can you make the mirror script available for hostong on JOSM server or whereever necessary?

Isn't this the mirroring script:
​https://github.com/openstreetmap/openstreetmap-mirror/blob/master/josm-mirror.sh

or this:
​https://github.com/simon04/josm-mirror/blob/master/josm-mirror.sh

(just by reading the history of this ticket)

[stoecker]

Fine. That's a start.

@Vincent: Can we move the mirror from openstreetmap to JOSM organization?

[stoecker]

Or maybe recreate it and already name it "josmmirror"?

[simon04]

Sorry, I overlooked the notification. Feel free to message me directly next time.

The mirroring broke due to (1) the switch from ​http://josm.openstreetmap.de/svn/trunk to https://josm.openstreetmap.de/svn/trunk and (2) the inclusion of windows/plugins/stdutils.

I updated the URL following ​https://stackoverflow.com/a/268767. The JOSM mirror is up again. I'll look into the josm-plugins mirror next.

[stoecker]

Could you please make the current mirror script and also the initial setup available (i.e. documentation). I'd still like to make the mirror from JOSM server, so that JOSM admins can fix issues.

[simon04]

The scripts plus all documentation I have is here: ​https://github.com/simon04/josm-mirror. Some additional information can be found in my comment:10.

[wiktorn]

Looks like mirror script stopped at [13698]

[simon04]

The mirror is up again. Due to an existing .git/index.lock, all mirroring Git commands failed.

[Don-vip]

The mirror branch works fine but it seems svn:externals fail for master.

[Don-vip]

When we switch to git this issue will finally be gone :)

[Don-vip]

Ticket #17937 has been marked as a duplicate of this ticket.

[Don-vip]

According to #17937 mirror stopped at May 25th.

[anonymous]

Hi, this issue is still not solved. The mirror was not updated since May 25th, unless I'm missing something?

[mkoniecz]

Hi, this issue is still not solved.

And that is why this issue remains open. All open issues are not solved, JOSM bug tracker has more than 1700 of them - https://josm.openstreetmap.de/query?status=assigned&status=needinfo&status=new&status=reopened&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&col=time&desc=1&order=id&report=4

If you want to help with some it is really welcomed, but useless +1 posts are useless and actually decrease chance of someone deciding to work on it.

The mirror was not updated since May 25th, unless I'm missing something?

See comment above yours.

[anonymous]

Thanks for the answer. How can I help working on this issue then? Is there any kind of log or error message I could look at to understand what's going on?

From what I see on ​https://github.com/openstreetmap/openstreetmap-mirror/blob/master/cronjobs, the josm mirroring is done with the svn-mirror script instead of the josm-mirror script. Is that expected? Also the script doesn't seem to be executable after cloning.

[anonymous]

I just found this: ​https://hub.docker.com/r/crazymax/svn2git-mirror Something like that wouldn't help?

[simon04]

I fixed the mirror again (the last time?):

• It now reflects all changes up to r16200 – ​https://github.com/openstreetmap/josm
• I dropped updating the mirror branch by resolving the svn:externals. This is error prone and no longer needed due to #16860. – ​https://github.com/simon04/josm-mirror/commit/868574e6588f26816827ed3ff1a4f498463c15c7
• I changed the default branch on GitHub to master

Please enjoy and let's dream of a future where JOSM's source code is hosted on Git only.

[Don-vip]

Thank you Simon :)

[simon04]

With the newly introduced git sparse-checkout one can clone a minimal JOSM git repository:

$ git clone --filter=blob:none --sparse https://github.com/openstreetmap/josm.git
$ cd josm/
$ git sparse-checkout add src nodist resources scripts tools

$ du -sh .; du -sh .git
164M    .
60M     .git

$ ant
...
BUILD SUCCESSFUL

[floscher]

​https://github.com/openstreetmap/josm-plugins was last updated at the end of 2017, see also ​https://github.com/openstreetmap/josm-plugins/pull/15

[simon04]

Fixed via #20126.