OSM Mapnik tiles slow to load (due to ImageIO#read slowness?)

[The111]

I apologize if this is not actually a true JMapViewer bug, but I'd like to at least open a discussion on this, and I'm not sure where else to do so. I contributed to JMapViewer a few times in the past, and it powers my application ​GPXCreator.

I don't know when exactly it happened, but OSM Mapnik tile loading seems way slower now than it used to be. I traced it down to ImageIO#read. There are lots of Google results for ImageIO being slow at png encoding in general. There are not many solutions, and none of them have helped me. The two most common ones are (1) disable disk caching in ImageIO and (2) install Java Advanced Imaging to leverage native image encoding capabilities on your platform. Again, neither seemed to make a difference.

I'm pretty sure the slowness is not the network IO, but the actual byte encoding. What's weirder is that it wasn't always like this. But looking back through older JMapViewer revisions, it seems ImageIO.read was always there. And ImageIO is not so slow for other tile sources, even ones with larger byte payloads like Bing sat images (which are admittedly jpg, not png). So, why were Mapnik tiles faster to encode in the past?
Has OSM changed their data format returned for HTTP requests, such that the encoding load is higher?

Or is something on my machine to blame? I did recently reimage my desktop with Win10. Can anybody else confirm that OSM Mapnik tiles seem far slower to load than the other sources, and that the slowness is not network IO?

As a sidenote, just as an experiment I even locally tried replacing ImageIO with Apache Commons Imaging, and it is equally slow. :-( This suggests that maybe it's not the fault of ImageIO, but either (a) my machine or (b) OSM payloads changing somehow.

[The111]

I was mistaken, it is indeed network IO slowness. In my test setup where I swapped to Apache Commons Imaging, I dug down deeper inside the source, and all the slowness is happening inside InputStream#read.

If I load a lot of tiles in busy map areas, anywhere from 1 to 5sec is spent reading from the http input stream for each tile.

Whereas browsing openstreetmap.org with Chrome, http requests to the same URLs complete in 100-500ms (10x faster). Maybe they are throttling our User-Agent and this is all by design? In which case I apologize for the noise.

EDIT: Nope, not related to the User-Agent string. So either this is still intentional throttling from OSM side somehow, or the Java input stream is slow.

[Don-vip]

Something changed on tile server side some weeks ago, related to User-Agent, see #17772.
I had to set a custom User agent in [o35020] to retrieve nominal behaviour. It seems the default Java User-Agent is blacklisted or something.

[The111]

Yeah, I'm using the version with your latest User-Agent. I also tried using a Chrome User-Agent. In both cases I get tiles, but slow. Do you observe that Mapnik tiles take way longer than Bing tiles to load, sometimes 3-5 seconds?

[Don-vip]

I didn't measure performances, I only made sure tiles were returned. What happens if you set JOSM user agent, is it faster?

[The111]

Yes, that is it for sure, thanks for the pointer in the right direction. I don't know much about the history of JOSM and User-Agent here, but a little bit of black box testing suggests that OSM server definitely whitelists "JOSM/.*" for much faster tile serving.

With User-Agent = ""
Tiles do not load at all, HTTP error code 429 (as you know).

With User-Agent = "JOSM" (or really any non-empty string)
Many tiles take longer than 1sec to load, the worst case is 4-5sec.

With User-Agent = "JOSM/"
No tile ever takes longer than 1sec, most are nearly instant.

FWIW, the actual User-Agent that JOSM compiled on my machine constructs is: "JOSM/1.5 (UNKNOWN en) Windows 10 64-Bit Java/1.8.0_212"

But I shortened it for my black box testing.

Should JMapViewer be updated to use JOSM style User-Agent?

[stoecker]

No. JOSM user agent is for JOSM only. JMapViewer outside JOSM needs other agent strings. Using it is fine for tests, but not more.

[stoecker]

BTW the UNKNOWN indicates the build process is unable to determine SVN version.

[stoecker]

For your case I'd set an proper GPXCreator/version (probably including JMapViewer and Java in the string) user agent and contact admins to accept that agent in the whitelist. False identification is usually never liked.

For JOSM server we had a case of a site scraper who switched to a valid JOSM user agent after the first blocks due to heavy page loading. That made blocking harder and me angry, but it did not prevent my reaction.

[The111]

Thanks stoecker. Do you know which admins specifically I should contact? Is it worth whitelisting JMapViewer agent in general?

Agreed entirely about falsifying ID, but it is interesting to me in my testing that even spoofing a Chrome User-Agent didn't seem to improve things (even though Chrome loads tiles fast from main OSM webapp), but when I tried the JOSM User-Agent is when things improved. I don't have any relationship with OSM people and I half suspect I won't get my wish for whitelisting granted, I imagine there is a reason they give JOSM special privileges, and I can't very well argue with them on that. Any more specific advice re: who and how to approach this is appreciated. And also still wondering if JMapViewer as a general library should be more privileged, or if I should just focus on my app. Thanks!

[stoecker]

Replying to The111:

Thanks stoecker. Do you know which admins specifically I should contact?

There is a contact section at ​https://operations.osmfoundation.org/policies/tiles/ linking to OSM wiki.

Is it worth whitelisting JMapViewer agent in general?

You can make them aware of the topic.

Agreed entirely about falsifying ID, but it is interesting to me in my testing that even spoofing a Chrome User-Agent didn't seem to improve things (even though Chrome loads tiles fast from main OSM webapp), but when I tried the JOSM User-Agent is when things improved. I don't have any relationship with OSM people and I half suspect I won't get my wish for whitelisting granted, I imagine there is a reason they give JOSM special privileges, and I can't very well argue with them on that. Any more specific advice re: who and how to approach this is appreciated.

I don't think they want to specifically block a valid use, but I'm pretty sure there are many bulk downloaders misusing open source components like JMapViewer and blocking these results in unwanted side effects. This also answers your question about JMapViewer. As this is a library, there may be good and bad tools using it. That makes it hard for generic rules :-)

In case of a known user agents it is also much easier to contact the author when the software goes amok. That helps a lot for whitelisting. Not all server floodings are caused by bad intent. Some are simply bugs.

And also still wondering if JMapViewer as a general library should be more privileged, or if I should just focus on my app. Thanks!

Focus on your app, but tell the admins the situation and let them handle JMapViewer as they think appropriate.

[The111]

Thanks for the info stoecker. I'll reach out to them soon. Closing this as it is clearly not a bug.

[Don-vip]

@The111 did you contact them? Feedback would be valuable to address #17991

[The111]

Sorry for the slow reply, I've been traveling a lot lately. I mostly decided I was ok with the slowish OSM tiles, I mainly use some other custom tile sources of my own anyway. So no, I have not contacted them. Also, I looked at #17991, and I'm not sure I can repro it. I checked out the latest from SVN and built the Demo, and it seemed to run fine for me.

[Don-vip]

No problem, thanks for the feedback. I can't reproduce it neither so I'll close this ticket.