support regional taginfo instances from Geofabrik

[Don-vip]

Follow-up of #

18302#comment:20:

Frederik just made a new version of GeoFabrik json index:
​http://download.geofabrik.de/index-v1-nogeom.json

It contains links to their regional Taginfo instances with ISO3166 ids so that we can map them with our internal boundaries.

I've set up a remote link on JOSM website:
https://josm.openstreetmap.de/remote/geofabrik-index-v1-nogeom.json

Geofabrik provides regional taginfo instances for all their extracts.

During the February 2020 hack week-end I worked with Frederik to provide an index matching our needs:
ticket:18302#comment:20

Now we need only to parse this index and provide links in the UI.

[Don-vip]

In 15876/josm:

fix #18729, see #18302 - support regional taginfo instances from Geofabrik

[simon04]

Out of the 417 entries, 192 entries cannot be used since they do not specify a ISO3166 code. Some of them are aggregated regions such as the DACH region, the Alps or Europe, some of them are sub-regions/states. Maybe none of them is strictly needed, however, statistics on the aggregated regions might be of interest?

https://taginfo.geofabrik.de/...
/africa/
/africa/canary-islands/
/africa/comores/
/africa/south-africa-and-lesotho/
/asia/
/asia/gcc-states/
/asia/japan/chubu/
/asia/japan/chugoku/
/asia/japan/hokkaido/
/asia/japan/kansai/
/asia/japan/kanto/
/asia/japan/kyushu/
/asia/japan/shikoku/
/asia/japan/tohoku/
/australia-oceania/
/central-america/
/central-america/bahamas/
/central-america/belize/
/central-america/cuba/
/central-america/haiti-and-domrep/
/central-america/jamaica/
/europe/
/europe/alps/
/europe/azores/
/europe/britain-and-ireland/
/europe/dach/
/europe/france/alsace/
/europe/france/aquitaine/
/europe/france/auvergne/
/europe/france/basse-normandie/
/europe/france/bourgogne/
/europe/france/bretagne/
/europe/france/centre/
/europe/france/champagne-ardenne/
/europe/france/corse/
/europe/france/franche-comte/
/europe/france/guadeloupe/
/europe/france/haute-normandie/
/europe/france/ile-de-france/
/europe/france/languedoc-roussillon/
/europe/france/limousin/
/europe/france/lorraine/
/europe/france/martinique/
/europe/france/mayotte/
/europe/france/midi-pyrenees/
/europe/france/nord-pas-de-calais/
/europe/france/pays-de-la-loire/
/europe/france/picardie/
/europe/france/poitou-charentes/
/europe/france/provence-alpes-cote-d-azur/
/europe/france/reunion/
/europe/france/rhone-alpes/
/europe/germany/baden-wuerttemberg/
/europe/germany/baden-wuerttemberg/freiburg-regbez/
/europe/germany/baden-wuerttemberg/karlsruhe-regbez/
/europe/germany/baden-wuerttemberg/stuttgart-regbez/
/europe/germany/baden-wuerttemberg/tuebingen-regbez/
/europe/germany/bayern/
/europe/germany/bayern/mittelfranken/
/europe/germany/bayern/niederbayern/
/europe/germany/bayern/oberbayern/
/europe/germany/bayern/oberfranken/
/europe/germany/bayern/oberpfalz/
/europe/germany/bayern/schwaben/
/europe/germany/bayern/unterfranken/
/europe/germany/berlin/
/europe/germany/brandenburg/
/europe/germany/bremen/
/europe/germany/hamburg/
/europe/germany/hessen/
/europe/germany/mecklenburg-vorpommern/
/europe/germany/niedersachsen/
/europe/germany/nordrhein-westfalen/
/europe/germany/nordrhein-westfalen/arnsberg-regbez/
/europe/germany/nordrhein-westfalen/detmold-regbez/
/europe/germany/nordrhein-westfalen/duesseldorf-regbez/
/europe/germany/nordrhein-westfalen/koeln-regbez/
/europe/germany/nordrhein-westfalen/muenster-regbez/
/europe/germany/rheinland-pfalz/
/europe/germany/saarland/
/europe/germany/sachsen-anhalt/
/europe/germany/sachsen/
/europe/germany/schleswig-holstein/
/europe/germany/thueringen/
/europe/great-britain/england/
/europe/great-britain/england/bedfordshire/
/europe/great-britain/england/berkshire/
/europe/great-britain/england/bristol/
/europe/great-britain/england/buckinghamshire/
/europe/great-britain/england/cambridgeshire/
/europe/great-britain/england/cheshire/
/europe/great-britain/england/cornwall/
/europe/great-britain/england/cumbria/
/europe/great-britain/england/derbyshire/
/europe/great-britain/england/devon/
/europe/great-britain/england/dorset/
/europe/great-britain/england/durham/
/europe/great-britain/england/east-sussex/
/europe/great-britain/england/east-yorkshire-with-hull/
/europe/great-britain/england/essex/
/europe/great-britain/england/gloucestershire/
/europe/great-britain/england/greater-london/
/europe/great-britain/england/greater-manchester/
/europe/great-britain/england/hampshire/
/europe/great-britain/england/herefordshire/
/europe/great-britain/england/hertfordshire/
/europe/great-britain/england/isle-of-wight/
/europe/great-britain/england/kent/
/europe/great-britain/england/lancashire/
/europe/great-britain/england/leicestershire/
/europe/great-britain/england/lincolnshire/
/europe/great-britain/england/london/enfield/
/europe/great-britain/england/merseyside/
/europe/great-britain/england/norfolk/
/europe/great-britain/england/north-yorkshire/
/europe/great-britain/england/northamptonshire/
/europe/great-britain/england/northumberland/
/europe/great-britain/england/nottinghamshire/
/europe/great-britain/england/oxfordshire/
/europe/great-britain/england/rutland/
/europe/great-britain/england/shropshire/
/europe/great-britain/england/somerset/
/europe/great-britain/england/south-yorkshire/
/europe/great-britain/england/staffordshire/
/europe/great-britain/england/suffolk/
/europe/great-britain/england/surrey/
/europe/great-britain/england/tyne-and-wear/
/europe/great-britain/england/warwickshire/
/europe/great-britain/england/west-midlands/
/europe/great-britain/england/west-sussex/
/europe/great-britain/england/west-yorkshire/
/europe/great-britain/england/wiltshire/
/europe/great-britain/england/worcestershire/
/europe/great-britain/scotland/
/europe/great-britain/wales/
/europe/isle-of-man/
/europe/italy/centro/
/europe/italy/isole/
/europe/italy/nord-est/
/europe/italy/nord-ovest/
/europe/italy/sud/
/europe/netherlands/drenthe/
/europe/netherlands/flevoland/
/europe/netherlands/friesland/
/europe/netherlands/gelderland/
/europe/netherlands/groningen/
/europe/netherlands/limburg/
/europe/netherlands/noord-brabant/
/europe/netherlands/noord-holland/
/europe/netherlands/overijssel/
/europe/netherlands/utrecht/
/europe/netherlands/zeeland/
/europe/netherlands/zuid-holland/
/europe/poland/dolnoslaskie/
/europe/poland/kujawsko-pomorskie/
/europe/poland/lodzkie/
/europe/poland/lubelskie/
/europe/poland/lubuskie/
/europe/poland/malopolskie/
/europe/poland/mazowieckie/
/europe/poland/opolskie/
/europe/poland/podkarpackie/
/europe/poland/podlaskie/
/europe/poland/pomorskie/
/europe/poland/slaskie/
/europe/poland/swietokrzyskie/
/europe/poland/warminsko-mazurskie/
/europe/poland/wielkopolskie/
/europe/poland/zachodniopomorskie/
/north-america/
/north-america/us-midwest/
/north-america/us-northeast/
/north-america/us-pacific/
/north-america/us-south/
/north-america/us-west/
/north-america/us/california/norcal/
/north-america/us/california/socal/
/russia/central-fed-district/
/russia/far-eastern-fed-district/
/russia/kaliningrad/
/russia/north-caucasus-fed-district/
/russia/northwestern-fed-district/
/russia/siberian-fed-district/
/russia/south-fed-district/
/russia/ural-fed-district/
/russia/volga-fed-district/
/south-america/
/south-america/brazil/centro-oeste/
/south-america/brazil/nordeste/
/south-america/brazil/norte/
/south-america/brazil/sudeste/
/south-america/brazil/sul/

[simon04]

In 15908/josm:

see #18729 - TaginfoRegionalInstance: use Stream API

[Don-vip]

Replying to simon04:
These entries exist in JOSM dataset. Frederik, can you please add following ISO codes?

/africa/canary-islands/            ISO3166-1:alpha2=IC / ISO3166-2=ES-CN
/africa/comores/                   ISO3166-1:alpha2=KM
/central-america/bahamas/          ISO3166-1:alpha2=BS
/central-america/belize/           ISO3166-1:alpha2=BZ
/central-america/cuba/             ISO3166-1:alpha2=CU
/central-america/haiti-and-domrep/ ISO3166-1:alpha2=HT,DO
/central-america/jamaica/          ISO3166-1:alpha2=JM
/europe/france/guadeloupe/         ISO3166-1:alpha2=GP / ISO3166-2=FR-GP
/europe/france/martinique/         ISO3166-1:alpha2=MQ / ISO3166-2=FR-MQ
/europe/france/mayotte/            ISO3166-1:alpha2=YT / ISO3166-2=FR-YT
/europe/france/reunion/            ISO3166-1:alpha2=RE / ISO3166-2=FR-RE
/europe/isle-of-man/               ISO3166-1:alpha2=IM
/russia/kaliningrad/               ISO3166-2=RU-KGD

[simon04]

In 15913/josm:

fix #18754, see #18729 - TaginfoRegionalInstance: fix deadlock

[Frederik]

Added the missing codes. @simon04 combined regions will currently only carry an ISO code if they are the smallest region to contain the country like in the "haiti-and-domrep" example. I think adding DE,AT,CH to the "dach" extract would be confusing?

[stoecker]

See #18772 - currently geofabrik serves a broken JSON file.

[Frederik]

fixed

[simon04]

Frederik, the file ​​http://download.geofabrik.de/index-v1-nogeom.json is broken again.

[simon04]

Frederik, the file ​https://download.geofabrik.de/index-v1-nogeom.json is broken again.

[simon04]

In 16196/josm:

see #18729, see 18772 - Add TerritoriesTestIT

[anonymous]

Really sorry. I am not sure why this happens occasionally but have now modified the script so that it at least detects the "invalid json" situation and keeps yesterday's file then...

[Klumbumbus]

Replying to simon04:

In 16196/josm:

see #18729, see 18772 - Add TerritoriesTestIT

It fails on Jenkins: https://josm.openstreetmap.de/jenkins/job/JOSM-Integration/lastCompletedBuild/jdk=JDK8/testReport/org.openstreetmap.josm.tools/TerritoriesTestIT/testUtilityClass/

[Don-vip]

Probably something needs to be done in PluginHandlerTestIT to make sure all plugins are unloaded at the end.

[simon04]

In 16272/josm:

see #18729 - Do not initialize external territories data for unit tests

[Klumbumbus]

TerritoriesTestIT still fails on Jenkins.

[taylor.smock]

There is an NPE if Territories.getRegionalTaginfoUrls is called during a test (it isn't called directly by my test code -- I'm using DataSet.setSelected).

I'll post some fixes for that. Not great fixes, but they'll avoid an NPE in tests.

See #19132 for the patches.