Trigger macOS re-build on github when changeset marked at tested

[Stereo]

This is a work in progress, intended to run on the josm server when releasing a new 'tested' version. Instead of triggering a new macOS build locally like in the past, this finds the github action workflow that contained git-svn-id: https://josm.openstreetmap.de/svn/trunk@$josm_revision in its commit message.

When a git action run is triggered, it fetches https://josm.openstreetmap.de/tested. If /tested has been properly updated before triggering the rebuild, it will build as a release instead of a pre-release, like ​https://github.com/grischard/josm/releases/tag/17329-tested

The shell script requires jq to parse the github json.

I've never seen what the build process looks like behind the curtain on the josm server - will this work? Comments and improvements very welcome.

#!/bin/bash

set -Eeuo pipefail
# Don't leak my tokens
set +x
# Don't glob
set -f

if [ -z "${1-}" ]; then
    echo "Usage: $0 josm_revision"
    echo "For example: $0 17329"
    exit 1
else
    josm_revision="${1-}"
fi

github_repo='openstreetmap/josm'
# Grab yours for free today exclusively at https://github.com/settings/tokens
github_oauth_token='invalidtoken'

if [[ "$josm_revision" == "$(curl --silent https://josm.openstreetmap.de/tested)" ]]; then
    echo "$josm_revision is indeed the current tested release."
else
    echo "Be careful: $josm_revision is not the current tested release at https://josm.openstreetmap.de/tested."
fi

# echo "Fetching re-run url for $josm_revision from past actions in $github_repo"

if rerun_url="$(
    curl \
        --silent \
        --fail \
        --header 'Accept: application/vnd.github.v3+json' \
        --header "Authorization: token $github_oauth_token" \
        --header 'Content-Type: application/json' \
        https://api.github.com/repos/$github_repo/actions/runs |
        jq \
            --raw-output \
            --arg JOSM_REVISION "$josm_revision" \
            '.["workflow_runs"] | .[] | select(.head_commit.message | contains("\n\ngit-svn-id: https://josm.openstreetmap.de/svn/trunk@" + $JOSM_REVISION)) | .rerun_url'
)"; then
    if [ -z "$rerun_url" ]; then
        echo >&2 "Could not get rerun_url for commit matching $1 in $github_repo. Have a look at https://api.github.com/repos/$github_repo/actions/runs"
        exit 1
    else
        # echo "I got $rerun_url which I will now request"
        if test "$(
            curl \
                --silent \
                --fail \
                -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url"
        )"; then
            echo "Re-launched the build for $josm_revision"
        else
            echo >&2 -n "Could not re-launch the build for $josm_revision in $github_repo. Message: "
            # Re-fetching with curl since --fail earlier doesn't let us capture output :/
            curl --silent -X POST \
                --header 'Accept: application/vnd.github.v3+json' \
                --header "Authorization: token $github_oauth_token" \
                "$rerun_url" | jq -r .message 1>&2
            exit 1
        fi
    fi
else
    echo >&2 "Could not get runs for git-svn-id $1 in $github_repo. Did you maybe push on top of it in git  ? Have a look at https://api.github.com/repos/$github_repo/actions/runs"
    exit 1
fi

[simon04]

Do we need to reimplement the git-svn-id parsing? It's already present in the Ant task init-git-revision-xml.

[Stereo]

@simon04 We don't have a git commit hash on that server at this point, unless I'm wrong. This is the best way I've found of uniquely identifying an svn revision from ​https://api.github.com/repos/openstreetmap/josm/actions/runs

[stoecker]

I've never seen what the build process looks like behind the curtain on the josm server - will this work?

I'll rewrite it into the used script.

Steps are:

• Run first URl
• Extract data
• Run second URL?

[Stereo]

Yeah, the first URL returns one rerun_url per run. We find the right svn revision ID in the first URL, and call that.

I was hoping it could live in /scripts so that it can be useful to others.

[stoecker]

Nah, that will be only a few lines of additional perl code in the existing script :-)

[stoecker]

I also need a way to test if build worked and what's the recent MacOS version available (i.e. a feedback).

[Stereo]

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

[stoecker]

Replying to Stereo:

And I fought myself so hard to resist writing it in Python! :)

It's possible to create a github action that triggers when a new release is published. I'll see if it can work as I expect it to.

No. Not in the release process. Something independent which delivers the "current state" version number, however it is. So that on the wiki page it get's visible when build failed and MacOS stays on old version. Ideally a HTTP request to a page which is easy parsable.

Otherwise, an apache redirect could always link to the most recent release - github has an url alias for that.

That would be fine. How is it?

[simon04]

We could also manually git tag (such as v20.11) the relevant Git commit with the version number after a release. In my local Git repository, I've tagged all 2020 releases this way (turned out to be very helpful for tracing back changes and using git bisect).

[Stereo]

For example, ​https://github.com/grischard/josm/releases/latest/download/JOSM-macOS-java15.zip redirects to ​https://github.com/grischard/josm/releases/download/17329-tested/JOSM-macOS-java15.zip . The wiki or apache could, when we have a proper release on the openstreetmap/josm github, just send people to ​https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip .

To get the mac version number for the wiki, you could parse the github release api for openstreetmap/josm. It would look similar to ​https://api.github.com/repos/grischard/josm/releases/latest and you could search for something like "tag_name": "17329-tested".

[simon04]

How does ​https://github.com/grischard/josm/releases/ differ from ​https://github.com/openstreetmap/josm/releases/?

[Stereo]

Oh, and I'd recommend not doing a release on openstreetmap/josm for 17329 because I did a couple of changes ( ​https://github.com/openstreetmap/josm/compare/master...grischard:master ) and it'd just be a mess.

@simon04 sorry, didn't see your comment before I posted. Those are the differences, to make releases work, after Don-vip explained to me how my understanding of josm releases was completely flawed :)

[Stereo]

​https://github.com/openstreetmap/josm/pull/67 shows the corresponding changes that need to be reflected on the josm side in a more stable, readable way.

[stoecker]

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

[Stereo]

@simon04 I like your idea and I'm wondering if you have an automatic way of telling that, say, 17239 is 20.11.

[simon04]

I've tagged my Git repository manually. I don't think that Git tags should be applied automatically. For many Git projects, the release workflows starts automatically after a manual (and possibly signed) Git tag.

[Don-vip]

Replying to stoecker:

@Vincent: Can you take care of this till the weekend. I'd like to implement the scripting stuff at the weekend.

Sure I'll take a look tonight

[Don-vip]

In 17373/josm:

see #20146 - Improvements to macOS build and github actions (patch by Stereo)

From ​https://github.com/openstreetmap/josm/pull/67:

• Separate the create-release github action to pass the release url to all subsequent jobs, to be able to...
• Upload .jar artefacts from all runs, and a separate .app for all java versions for macOS
• Better caching for ant/junit
• Support for the release workflow from #20146
• A few cosmetic fixes, more readable syntax, etc.

[Stereo]

@stoeker do you maybe want to do a test release together an evening this week?

[stoecker]

Replying to Stereo:

@stoeker do you maybe want to do a test release together an evening this week?

ATM I feel more like reading than programming. So probably it will have to wait till next week.

[stoecker]

Sorry for the delays. It's not forgotten. If possible please provide a new link for the current release until I fix this ticket. I'm a bit demotivated to do much programming currently and instead am reading many books.

[Stereo]

Don-vip has updated the redirect for now.

Don't let the grey weather get you down, and enjoy your reading.

[Stereo]

Ticket #20327 has been marked as a duplicate of this ticket.

[Stereo]

This has been ready to go since November. Can it be put into production this month?

[stoecker]

Milestone renamed

[stoecker]

Sorry. This time I simply forgot. I had a look now. What do I do when there are more than one run. Use the newest?

Check for 17489
Found 17489 https://api.github.com/repos/openstreetmap/josm/actions/runs/553936406/rerun
Found 17489 https://api.github.com/repos/openstreetmap/josm/actions/runs/553936403/rerun

use JSON;

sub triggergithub($) {
  my ($version) = @_;
  my $cmd = "curl --silent --fail --header 'Accept: application/vnd.github.v3+json'"
  . " --header 'Authorization: token ....'"
  . " --header 'Content-Type: application/json'"
  . " https://api.github.com/repos/openstreetmap/josm/actions/runs";
  my $d = decode_json(`$cmd`);
  print "Check for $version\n";
  for my $run (@{$d->{workflow_runs}})
  {
    if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version /)
    {
      print "Found $version $run->{rerun_url}\n";
    }
  }
}

[Stereo]

Good to hear from you again :)

Now I'm really wondering why we'd have two runs for 17489 - did you maybe trigger another test run already? The runs are supposed to run once per commit, and maybe a second time if that commit gets blessed by you to a tested release.

In any case, the latest run should be a reasonable assumption. The code should be the same anyway.

[stoecker]

Replying to Stereo:

Good to hear from you again :)

Now I'm really wondering why we'd have two runs for 17489 - did you maybe trigger another test run already? The runs are supposed to run once per commit, and maybe a second time if that commit gets blessed by you to a tested release.

Happens since the recent changes:

17491 17491
17489 17489
17488 17488
17487 17487
17486
17484
17483
17482
17476
17475
17474
17472
17471

[Stereo]

Ahh, one of them is probably for the CodeQL thing which I added. Damn it, Guillaume.

You can distinguish them by name, where we have "name": "Java CI" or "name": "CodeQL"

Something like this? (untested)

    if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version / && $run->{name} == "Java CI") 

[stoecker]

Replying to Stereo:

... && $run->{name} == "Java CI")

Nearly. String compare needs "eq" in perl otherwise it is a numeric compare :-)

Try

perl -e 'print "true\n" if "Hallo" == "Ballo"'

And to be fair to perl later try :-)

perl -w -e 'print "true\n" if "Hallo" == "Ballo"'

[Stereo]

How can I run the final perl code with my repo token to see if it correctly triggers a rebuild?

[stoecker]

Finishing

#!/usr/bin/perl -w
use JSON;
use strict;

sub triggergithub($) {
  my ($version) = @_;
  eval
  {
    my $token = "...";
    my $curlbase = "curl --silent --max-time 30"
    . " --header 'Authorization: token $token'";
    my $cmd = "$curlbase --fail https://api.github.com/repos/openstreetmap/josm/actions/runs";
    my $d = decode_json(`$cmd`);
    for my $run (@{$d->{workflow_runs}})
    {
      if($run->{head_commit}{message} =~ /git-svn-id: https:\/\/josm\.openstreetmap\.de\/svn\/trunk\@$version / && $run->{name} eq "Java CI")
      {
        my $url = $run->{rerun_url};
        print "Found $version $run->{rerun_url}\n";
        die "Strange URL $url" if $url !~ /^https:\/\/api\.github\.com\/[a-z0-9\/]+$/; # prevent security risks
        my $cmd = "$curlbase -X POST $url";
print "$cmd\n";
print `$cmd`;
      }
    }
  };
  print STDERR "Error rebuilding MacOS: $@" if $@;
}

my $ver = 17489;
print "Check for $ver\n";
triggergithub($ver);

I always get

  "message": "Must have admin rights to Repository.",
  "documentation_url": "https://docs.github.com/rest/reference/actions#re-run-a-workflow"

I added "repo", but even tried all, but it does not help.

How can I run the final perl code with my repo token to see if it correctly triggers a rebuild?

Save into file, insert the token and call "perl file".

[Stereo]

Hmm, I get a strange error - it seems to choke on the json somehow:

> perl foo.pl
Check for 17489
Error rebuilding MacOS: , or } expected while parsing object/hash, at character offset 2 (before "(end of string)") at tagati.pl line 13.

But by adapting my shell script to only run on "Java CI", I can reproduce your problem by using my token on the openstreetmap/josm repo, where I don't have full access.

Do you have both 'repo' and 'workflow' checked in the token scopes?

[stoecker]

I tried different setups. Once even with everything enabled. Are we maybe missing another setting in JOSM project?

I'll generate a new token. Maybe that helps.

For your error do comment the eval and closing bracket. Then you get a more descriptive error. I'd assume the JSON is somehow invalid. Maybe curl call failed?

[Stereo]

You *are* admin, right? :)

[stoecker]

Replying to Stereo:

You *are* admin, right? :)

Well. No :-)

Aaargh. Seems I/we never added the relevant user to the openstreetmap group.

[stoecker]

So. Copied into the relevant job. Should be called with next tested. Auto-Updating the link still missing as well as some cleanup (removing old mac stuff).

[Stereo]

Fingers crossed. Thank you!

For the links, #20046 can be done and closed as soon as there's a new stable build at ​https://github.com/openstreetmap/josm/releases/latest/download/JOSM-macOS-java15.zip . Whether we do that through a wiki search/replace or an Apache config update is up to you :).