[Patch] Detect preset wiki languages automatically

[simon04]

Instead of maintaining various OSM wiki languages in defaultpresets.xml, maintaining presets_lang.pl, running presets_lang.pl from time to time, still applying manual changes (cf. #17171), let JOSM detect the wiki language automatically. This features has been implemented long ago in HelpAction.displayHelp: it opens https://wiki.openstreetmap.org/wiki/[lang]:[page] if it exists and falls back to https://wiki.openstreetmap.org/wiki/[page].

So in a nutshell …

data/defaultpresets.xml

@@ -326 +326 @@
         <reference ref="voltage_frequency" />
     </chunk>
     <chunk id="power_generator">
-        <link href="https://wiki.openstreetmap.org/wiki/Tag:power=generator"
-                  cs.href="https://wiki.openstreetmap.org/wiki/Cs:Tag:power=generator"
-                  de.href="https://wiki.openstreetmap.org/wiki/DE:Tag:power=generator"
-                  es.href="https://wiki.openstreetmap.org/wiki/ES:Tag:power=generator"
-                  fr.href="https://wiki.openstreetmap.org/wiki/FR:Tag:power=generator"
-                  it.href="https://wiki.openstreetmap.org/wiki/IT:Tag:power=generator"
-                  ja.href="https://wiki.openstreetmap.org/wiki/JA:Tag:power=generator"
-                  pl.href="https://wiki.openstreetmap.org/wiki/Pl:Tag:power=generator"
-                  pt.href="https://wiki.openstreetmap.org/wiki/Pt:Tag:power=generator"
-                  ru.href="https://wiki.openstreetmap.org/wiki/RU:Tag:power=generator" />
+        <link wiki="Tag:power=generator" />
         <space />
         <key key="power" value="generator" />
         <reference ref="name_ref_operator" />

In total: 719 insertions(+), 5651 deletions(-)

Relates to this discussion at josm-dev@: ​https://lists.openstreetmap.org/pipermail/josm-dev/2018-November/008171.html (Add templated version of wiki href to link element in presets)

[stoecker]

It's not so easy. You overlooked all these cases where the links don't have the same name.

[naoliv]

For pt_BR it will be necessary to consider the language as pt, since pt_BR was unified into pt (ie, people using JOSM in pt or pt_BR should open the same Portuguese wiki pages).
With the patch (actually after [14617]), using pt_BR only opens the English pages.

[simon04]

Replying to stoecker:

It's not so easy. You overlooked all these cases where the links don't have the same name.

Please name one. I only stumbled over ​https://wiki.openstreetmap.org/wiki/Link_(highway vs. ​https://wiki.openstreetmap.org/wiki/Cs:Highway_link, but the former redirects to ​https://wiki.openstreetmap.org/wiki/Highway_link. IMO we could accept 0.1% non-detected language links in favour of simplicity / elegance / reduced code / reduced maintenance burden.

Replying to naoliv:

For pt_BR it will be necessary to consider the language as pt, since pt_BR was unified into pt (ie, people using JOSM in pt or pt_BR should open the same Portuguese wiki pages).
With the patch (actually after [14617]), using pt_BR only opens the English pages.

Good to know… We should reflect the merge in LanguageInfo. The merge has taken place on the OSM wiki, the JOSM wiki, the JOSM i18n?

[naoliv]

Replying to simon04:

Good to know… We should reflect the merge in LanguageInfo. The merge has taken place on the OSM wiki, the JOSM wiki, the JOSM i18n?

Only OSM wiki was unified. The rest are still (and I guess will stay) different.

[stoecker]

Replying to simon04:

Replying to stoecker:

It's not so easy. You overlooked all these cases where the links don't have the same name.

Please name one. I only stumbled over ​https://wiki.openstreetmap.org/wiki/Link_(highway vs. ​https://wiki.openstreetmap.org/wiki/Cs:Highway_link, but the former redirects to ​https://wiki.openstreetmap.org/wiki/Highway_link. IMO we could accept 0.1% non-detected language links in favour of simplicity / elegance / reduced code / reduced maintenance burden.

Hmm, seems the wiki situation changed. E.g. Link_(highway) or Highway_link actually has references to "Pt:Tag:highway=motorway_link", but this is missing totally now. Also the wiki does not refernce redirects in the language pages. Maybe they broke the wiki in that instance? That would mean that your approach would actually be better, as the links then work again.

Can you please extract the code changes and attach them separately for review?

Replying to naoliv:

For pt_BR it will be necessary to consider the language as pt, since pt_BR was unified into pt (ie, people using JOSM in pt or pt_BR should open the same Portuguese wiki pages).
With the patch (actually after [14617]), using pt_BR only opens the English pages.

Good to know… We should reflect the merge in LanguageInfo. The merge has taken place on the OSM wiki, the JOSM wiki, the JOSM i18n?

JOSM wiki and JOSM i18n is not affected. That would only be the case, when pt_BR translation gets dropped. But we need to change JOSM code, that the OSM wiki prefix for pt_BR is "Pt:".

Note that JOSM wiki uses language code derived rules, whereas the prefix for OSM wiki seems to be manually decided. They are not identical.

[stoecker]

Ticket #17171 has been marked as a duplicate of this ticket.

[stoecker]

If we go this way we need to:

• test that the added English links actually refer to non-redirect pages
• have a way to test redirects later (i.e. modify the wiki scraper)

If not:

• update the wiki scraper to handle redirects correct

[simon04]

In 14646/josm:

fix #17173 - Detect preset wiki languages automatically

[simon04]

In 14647/josm:

see #17173 - Use correct language prefixes for OSM wiki

Reference: ​https://wiki.openstreetmap.org/wiki/Template:Languages

[Don-vip]

LinkTest fails.

[simon04]

In 14649/josm:

see #17173 - Fix LinkTest

[stoecker]

You missed to address the comment:8, the wiki documentation is missing and this

else if (type == LocaleType.OSM_WIKI) {
  return locale.getLanguage() + ':';
}

causes lowercase results which produce a redirect always. It should not be default to produce useless redirects for each wiki call. We should use a correct uppercase/lowercase table instead.

[Don-vip]

Replying to stoecker:

It should not be default to produce useless redirects for each wiki call.

We should add an integration test to check this.

[stoecker]

Replying to Don-vip:

Replying to stoecker:

It should not be default to produce useless redirects for each wiki call.

We should add an integration test to check this.

Sidenote: We must be careful with remote integration tests. E.g. We are currently accessing all our map services 4 times a day with our JOSM user agent. I'm not sure this is a good idea. Probably that should be reduced to once a day and also use another user agent.

[Don-vip]

Replying to stoecker:

I'm not sure this is a good idea.

Why?

[stoecker]

Replying to Don-vip:

Replying to stoecker:

I'm not sure this is a good idea.

Why?

Because as administrator I tend to block user agents and servers which constantly access the same stuff. Algorithmic data access is usually not caused by humans and thus unwanted data access. Other operators handle that similar.

Actually for automatic access we'd need to follow robots.txt protocols and thus be restricted.

If some operator decides to block JOSM user agent because of this that would be a VERY unwanted effect.

[Don-vip]

OK.

The i18n build is broken:

     [exec] /* unparsed line 7481             <link wiki="Relations/Proposed/Bridges_and_Tunnels" /> */

BUILD FAILED
/var/lib/jenkins/jobs/JOSM-i18n/workspace/i18n/build.xml:29: exec returned: 20

[simon04]

Replying to stoecker:

You missed to address the comment:8

I checked my replacements of href with wiki by hand: Except for Highway_link (see above), all localized hrefs only differed by their language prefix. Thus, I consider the probability of localized pages redirecting anywhere differnt very low.

the wiki documentation is missing

Done: TaggingPresets

causes lowercase results which produce a redirect always. It should not be default to produce useless redirects for each wiki call. We should use a correct uppercase/lowercase table instead.

This is (a) not a problem and (b) not new: (a) ​https://wiki.openstreetmap.org/wiki/de:key:name automatically resolves to ​https://wiki.openstreetmap.org/wiki/DE:Key:name not involving a redirect. (b) from org.openstreetmap.josm.gui.dialogs.properties.HelpAction we gladly opened De: beforehand.

Replying to Don-vip:

The i18n build is broken:

Thanks for noticing! Fixed in [o34809].

[stoecker]

Replying to simon04:

Replying to stoecker:

You missed to address the comment:8

I checked my replacements of href with wiki by hand: Except for Highway_link (see above), all localized hrefs only differed by their language prefix. Thus, I consider the probability of localized pages redirecting anywhere differnt very low.

As said in #17171 the script probably didn't handle redirects well. Thus I assume our links contain a lot of redirects or there wouldn't have been so many pt_BR remaining for manual fixing. This must be checked now (and later automatically) and cleaned. Redirects should only remain in seldom cases.

causes lowercase results which produce a redirect always. It should not be default to produce useless redirects for each wiki call. We should use a correct uppercase/lowercase table instead.

This is (a) not a problem and

Well, as said it is bad default behaviour.

(a) ​https://wiki.openstreetmap.org/wiki/de:key:name automatically resolves to ​https://wiki.openstreetmap.org/wiki/DE:Key:name not involving a redirect.

Check the data transmission yourself. It involves always a 301 redirect. If the URL line changes in a browser there is no way except a redirect to reach that goal (or JavaScript quirks).

(b) from org.openstreetmap.josm.gui.dialogs.properties.HelpAction we gladly opened De: beforehand.

Which tells me, that the existing code was not properly tested. Still that's no reason to leave it that way.

[stoecker]

In 14660/josm:

see #17173 - proper prefixes for OSM wiki

[stoecker]

Replying to stoecker:

In 14660/josm:

see #17173 - proper prefixes for OSM wiki

I don't have time to test each case ATM. Hope this is ok as is. 😇

[simon04]

Replying to stoecker:

I don't have time to test each case ATM. Hope this is ok as is. 😇

The unit tests need to be adapted: https://josm.openstreetmap.de/jenkins/job/JOSM/4936/jdk=JDK8/testReport/org.openstreetmap.josm.tools/LanguageInfoTest/getWikiLanguagePrefix/

[stoecker]

In 14664/josm:

fix test cases, see #17173

[anonymous]

The information=map preset points to the Key:information page instead of the Tag:information=map page on the wiki.

[Klumbumbus]

In 14703/josm:

see #17173 - improve preset wiki links

[simon04]

I opened #17240 for the outstanding task.