Heads up: Maxar implementing new anti-scraping techniques

[Stereo]

​https://www.openstreetmap.org/user/@kevin_bullock/diary/391652

[Boggedy]

This may also require an auto expire feature for the JOSM imagery cache where Maxar and ESRI tiles are all autodeleted on exit.

[Don-vip]

Ticket #18473 has been marked as a duplicate of this ticket.

[Don-vip]

Ticket #18516 has been marked as a duplicate of this ticket.

[Don-vip]

See https://josm.openstreetmap.de/ticket/18473#comment:3 comments about the removal from default entries

[Don-vip]

I've received an e-mail from Maxar providing us (JOSM) a new API key to restore access to Maxar imagery. Their only condition is to not make the API key public on the wiki or the source code.

I already began related work in #17744 / [o35017] when the Bing API key was unavailable. I'll work on this tomorrow during GeoFabrik hack week-end.

[stoecker]

If the key should not be in the wiki or the source, where should it be?

[Don-vip]

I was thinking about providing it at build time and storing it somewhat in JOSM jar. Not completely hidden, but at least not easily crawlable. They seem ok with this.

[stoecker]

That's not possible. It would be a license violation of JOSM GPL.

[Don-vip]

Ok. Does it work if JOSM retrieves the key from JOSM website using a new endpoint? I'm thinking of a POST request with a user-agent check.

[stoecker]

That would work.

[stoecker]

If so, please make it expandable to other JOSM specific keys.

[Don-vip]

In 15853/josm:

see #18440 - non-blocking warning in case of invalid TMS entries

[Don-vip]

JMapViewer updated in [o35320].

[Don-vip]

Replying to stoecker:

If so, please make it expandable to other JOSM specific keys.

What other keys do you have in mind?

[stoecker]

Replying to Don-vip:

Replying to stoecker:

If so, please make it expandable to other JOSM specific keys.

What other keys do you have in mind?

We have a few other JOSM specific map keys. If we setup such a feature it should be possible to support other services as well. It should not be Maxar only.

[stoecker]

E.g.

​https://josm.openstreetmap.org/mapkey?id=maxar

returns the maxar key and ...

[SimonPoole]

@stoecker the GPL in general does not apply to configuration.

[stoecker]

Replying to SimonPoole:

@stoecker the GPL in general does not apply to configuration.

That's not configuration. GPL applies to sources and the tools/toolchain to build an executable from this. So there cannot be any secret in the build process. The code signing we do for JOSM is already a border case.

[Don-vip]

Replying to stoecker:

If we setup such a feature it should be possible to support other services as well. It should not be Maxar only.

No problem, this is what I had in mind.
EDIT: These are the entries I found with an API key:

• Bing
• City_of_Melbourne_Feb_2019
• City_of_Melbourne_May_2018
• geoimage.at
• geovekst-nib
• IBGE_DF_Addresses
• IBGE_Nomes_Ruas
• IBGE_Salvador_Streets
• lantmateriet-orto1960
• lantmateriet-orto1975
• lantmateriet-topowebb
• LINZ_NZ_Aerial_Imagery
• LINZ_NZ_Topo50_Gridless_Maps
• Mapbox
• mapbox_locator_overlay
• Maxar-Premium
• Maxar-Standard
• New_and_Misaligned_TIGER_Roads-2013
• opencylemap
• tf-landscape
• US_Forest_Service_roads_overlay
• US-TIGER-Roads-2017
• US-TIGER-Roads-2018
• US-TIGER-Roads-2019
• WSDOT-Roads-2017

Any missing?

[Don-vip]

In 15855/josm:

see #18440 - Fetch imagery API keys from JOSM website. Restore access to Maxar imagery. Requires JMapViewer 2.13

[anonymous]

Replying to stoecker:

Replying to SimonPoole:

@stoecker the GPL in general does not apply to configuration.

That's not configuration. GPL applies to sources and the tools/toolchain to build an executable from this. So there cannot be any secret in the build process. The code signing we do for JOSM is already a border case.

This not even the interpretation of the FSF.

[stoecker]

The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable.

I don't care about any "interpretation", especially given by an anonymous commenter. I follow the GPL texts and these are very clear in this fact.

[Don-vip]

In 15868/josm:

see #18440 - cache API keys in memory

[Don-vip]

Done for Maxar. Follow-up in #18726

[Don-vip]

In 15871/josm:

see #18440 - register API key provider in integration test

[stoecker]

Could you change the pattern to lowercase {apikey}? that's not a Java variable name.

[Don-vip]

Replying to stoecker:

Could you change the pattern to lowercase {apikey}? that's not a Java variable name.

Done in [o35329].

[Klumbumbus]

There are at least two ways to display the connectId in JOSM in cleartext. Is that a problem?

[stoecker]

Replying to Klumbumbus:

There are at least two ways to display the connectId in JOSM in cleartext. Is that a problem?

Hopefully not. Current solution is the best we can do in an OpenSource software.

[Don-vip]

Replying to Klumbumbus:

There are at least two ways to display the connectId in JOSM in cleartext. Is that a problem?

No, I warned them about this and they're ok. They just don't want the key to be too easily scraped (using a search engine for example).