Add missing google domains to blacklist

[pnorman]

For reference, ​http://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/io/OsmApi.java#L242 is where the current list is.

We are not currently blocking google.ru, google.pl, and googleapis.com, all of which have seen use.

".*\\.google(apis)?\\.[^/]*/.*" might be a better regex than adding them all individually, but it'd need checking as I'm not 100% sure everything is escaped right.

I also get 173.194.33.0-14, 173.194.41.128-142, 74.125.228.0-14 as IP ranges

[Don-vip]

The proper solution should be to define imagery blacklist in OSM API capabilities (as described in r3934, see ​http://josm.openstreetmap.de/ticket/6037#comment:11).

Can you ask to API developers if this is planned ?

[pnorman]

I don't believe anyone has planned adding a blacklist to /capabilities, nor do I think this was ever planned.

[Don-vip]

@team: do you remember if it was really planned on server side ? If not we should ask for it :)

[Don-vip]

I have created this ticket: ​https://trac.openstreetmap.org/ticket/5024

I hope it will be accepted...

[Don-vip]

Good thing: the ticket has not be closed as wontfix in the 5 minutes of its creation like I often see on trac.osm.org when we request something.

Bad thing: no response at all, I don't know if it's the good way to ask for it. If it's something that you really want at DWG, I'm sure you know how to make it happen, the ball is in your court :)

[Don-vip]

Well, this doesn't look like so major.

[pnorman]

I've raised the issue of if clients should be responsible for blocking imagery or if it should come from the server with the LWG, since there may be legal implications, but we've not met over Christmas.

[Don-vip]

Ok thanks for update. I assume it will be discussed in next minute, here ? ​http://www.osmfoundation.org/wiki/Working_Group_Minutes#Licensing_Working_Group

[Don-vip]

Is the LWG still alive?

[Don-vip]

Hello, I finally see ​some activity from Feb 25th stating:

Getting JOSM’s blacklist update is important for the DWG, be it by modifying the API to return a blacklist or not. The question is, should this be delivered by the OpenStreetMap API? The positive is that it actively helps prevent well meaning map data and editor software contributors from contaminating the OSM database with data from non-allowed sources. The potential negative to this is that it may imply that other source not on the blacklist are OK. After discussion, it was felt that the right balance could be struck by additionally adding a whitelist, which may also be of use in its own right. Therefore:

LWG recommends and supports an API-delivered blacklist AND whitelist. We urge that it be made clear in supporting documentation that if a source is not on the blacklist, that does not imply that it OK and research should be done before adding it.

Potential whitelist ​https://github.com/osmlab/editor-imagery-index

The next meeting was scheduled on March 25th but I can't find the minutes?

[stoecker]

A note: I won't support a whitelist. Especially not the mentioned one. But actually this is also nothing they should care about at all.

If the API supplies a reasonable blacklist we'll use it, but editor development is still our task and not that of a licensing working group.

[bastiK]

Replying to stoecker:

A note: I won't support a whitelist. Especially not the mentioned one. But actually this is also nothing they should care about at all.

If the API supplies a reasonable blacklist we'll use it, but editor development is still our task and not that of a licensing working group.

It's a good thing if the licensing working group checks the usage terms of imagery providers and compiles a list of "officially" approved servers. It would give confidence to the users that the work put into tracing stuff is not wasted. Everything not on the whitelist or the blacklist is simply not yet evaluated by the LWG. So as I understand this suggestion, it would be an additional service and by no means a restriction.

[stoecker]

So as I understand this suggestion, it would be an additional service and by no means a restriction.

I already told Frederik when he added the blacklist, that this in principle cannot work. It is a compromise to prevent the most obvious misuse. Otherwise we still need to trust our users. A white-list is too much. I'll never agree to it.

Think of something like HOT. Should they wait weeks or months until LWG approves their imagery? Would make the project totally useless.

I anyway want to introduce a verfied="date" tag to our maps list, so outdated entries get easier to spot. If LWG wants, they can also provide a list of verified URL's and we simply add an lwg_approved="date" to the entries and display that to the user. But not more. If they do so - will LWG be liable for wrong entries: Now we tell the user always "you are responsible"? A simple API list will not be enough in any case. It must be a more official document of the type "The LWG states:".

[anonymous]

I'm not quite sure if the problem is just one of wording (aka using the term "whitelist"), as was already said, the idea is that it should be a list of imagery sources that have already had a number of eyeballs on it and where there is a certain level of confidence that the sources are ok for use in OSM. It was never intended to block custom imagery, and in fact it has been stated multiple times that naturally custom imagery should still be possible, just that you need to check the suitability for OSM use on your "own". Or to put it differently, it tries to answer the question for developers of a new editor, which imagery sources can I use without needing to check each one myself?

As to not implementing a positive list: de facto JOSM already does so from an user perspective, just as most of the other editors, the pre-configured imagery sources already have an implied suitability for use in OSM.

At this point in time both the black list and a potential positive list are simply an offer of support and have not been declared mandatory (has anything ever been declared mandatory wrt editors?). It would just seem that supporting a black list through the API would enable all to react faster to ongoing misuse as it happens.

[Don-vip]

Any news on this subject from working groups side?

[pnorman]

See ​https://trac.openstreetmap.org/ticket/5024.

I don't see any reason to hold off updating the in-JOSM list while waiting for new API functionality.

[bastiK]

Blacklist is up:

$ wget -O- http://api.openstreetmap.org/api/capabilities

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="OpenStreetMap server" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
  <api>
    <version minimum="0.6" maximum="0.6"/>
    <area maximum="0.25"/>
    <tracepoints per_page="5000"/>
    <waynodes maximum="2000"/>
    <changesets maximum_elements="50000"/>
    <timeout seconds="300"/>
    <status database="online" api="online" gpx="online"/>
  </api>
  <policy>
    <imagery>
      <blacklist regex=".*\.googleapis\.com/.*"/>
      <blacklist regex=".*\.google\.com/.*"/>
      <blacklist regex=".*\.google\.ru/.*"/>
    </imagery>
  </policy>
</osm>

From our code (OsmApi.java) I get that the JOSM internal blacklist is now ignored:

            /* This is an interim solution for openstreetmap.org not currently
             * transmitting their imagery blacklist in the capabilities call.
             * remove this as soon as openstreetmap.org adds blacklists.
             * If you want to update this list, please ask for update of
             * http://trac.openstreetmap.org/ticket/5024
             * This list should not be maintained by each OSM editor (see #9210) */
            if (this.serverUrl.matches(".*openstreetmap.org/api.*") && capabilities.getImageryBlacklist().isEmpty()) {
                capabilities.put("blacklist", "regex", ".*\\.google\\.com/.*");
                capabilities.put("blacklist", "regex", ".*209\\.85\\.2\\d\\d.*");
                capabilities.put("blacklist", "regex", ".*209\\.85\\.1[3-9]\\d.*");
                capabilities.put("blacklist", "regex", ".*209\\.85\\.12[89].*");
            }

Who is responsible for maintaining the blacklist, so we can request to add the IP patterns?

[stoecker]

In 11532/josm:

see #9210 - remove interim imagery blacklist solution, the api now has a proper system

[stoecker]

Hmm. Really? Claiming a ticket closed 2 years ago?

[Don-vip]

well you cleaned up the code :)

[Don-vip]

My bad didn't see it was already closed.