Support for image/webp for WMTS satellite imagery?

[SanderH]

I'm trying to get a satellite imagery working that's available in the netherlands since a few days but don't get the WMTS version working.
Configured the WMTS imagery for this url:
​https://geodata.nationaalgeoregister.nl/luchtfoto/wmts?&request=GetCapabilities&service=WMS
The provider prefers consumers to use WMTS over WMS (which is working fine) to reduce load on their servers.

When selecting the WMTS from the Imagery menu, I get a nice box to choose from the 3 projections for the 2 layers.
Whatever you choose there it won't work. JOSM keeps complaining "Could not load image file from server".

The tile properties show for example this url with http status code 200:
​http://geodata.nationaalgeoregister.nl/luchtfoto/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=Actueel_ortho25&STYLE=default&FORMAT=image/webp&tileMatrixSet=EPSG:28992&tileMatrix=8&tileRow=134&tileCol=131

I noticed the default format is image/webp, which I'm unable to change to one of the known working image formats (png, png8, jpg) which is also supported for this WMTS.

Since everything else seems working OK (url does work in the browser, status code 200) the only thing I can think of is that the image format itself could be the problem.
Do you know if JOSM supports image/webp?

Can you have a look and see what's going wrong and maybe fix it accordingly if it's something JOSM can do about it?

Thanks in advance.

[Don-vip]

WebP is not supported by the JDK, see ​javabug:8116096. We must make sure to use only supported formats.

[Don-vip]

In 11559/josm:

fix #14361 - WMTS: download only supported image formats

[Stereo]

Would ​https://github.com/opsmatters/webp-imageio-core be an option?

[Don-vip]

In source program, coders need to put native lib files like .so/.dll/.dylib into the folder of java.library.path.

No go, sorry.

[anonymous]

​https://github.com/haraldk/TwelveMonkeys could be an option now?

[Sreeram K]

Reopening to start a discussion..

[taylor.smock]

I'll be honest, this kind of sounds like it could be done in a plugin (and probably should be done in a plugin). I don't know if JOSM really wants to bundle a bunch of jar files it doesn't directly depend upon. The few jars I checked from TwelveMonkeys seemed to be reasonably sized (<100kb).

[Sreeram K <kandimalla.sreeram@…>]

I made an attempt at ​https://github.com/ramSeraph/josm-webp-plugin , did not work. I am not very familiar with java. it almost feels like the imageio available to the imagery code was already initialised by the time the plugins load.

[taylor.smock]

Replying to Sreeram K <kandimalla.sreeram@…>:

I made an attempt at ​https://github.com/ramSeraph/josm-webp-plugin , did not work. I am not very familiar with java. it almost feels like the imageio available to the imagery code was already initialised by the time the plugins load.

Notes:

• I probably would have called it josm-imageio-extensions or something like that, and then downloaded common extensions from the maven repository (example: avif/heif in the future).
  • Use the plugin manifest Class-Path instead of packing the imageio extensions into the jar file if you do this.
  • This is more complicated. You probably don't want to do that right now.
• To get the current plugin to build, I would do this: git mv lib/build.gradle.kts .; git mv lib/src . in the root directory of the repository.
• Apply the following patch to get the WebP ImageIO extension registered on plugin load

  src/main/java/org/openstreetmap/josm/plugins/webp/WebpPlugin.java

  @@ -1 +1 @@
   package org.openstreetmap.josm.plugins.webp;
   
  +import javax.imageio.ImageIO;
   import org.openstreetmap.josm.plugins.Plugin;
   import org.openstreetmap.josm.plugins.PluginInformation;
   
  @@ -11 +12 @@
      */
     public WebpPlugin(PluginInformation info) {
        super(info);
  +     ImageIO.scanForPlugins();
     }
   }

[Sreeram K <kandimalla.sreeram@…>]

The plugin in the current form builds.. though I would definitely want to get rid of that extra sub directory structure.

And apparently the problem was not with the plugin, but with the webp layer I was testing.. I checked with another webp layer I found on the internet.. it seems to be working fine. even without the changes you suggested. I would like to debug further as to what is going wrong with the layer I am trying.

About the suggestion regarding the Class-Path and the name, would this be a prerequisite for adding it into the main plugin repository?

[taylor.smock]

Replying to Sreeram K <kandimalla.sreeram@…>:

The plugin in the current form builds.. though I would definitely want to get rid of that extra sub directory structure.

You probably have to run gradle inside the lib directory instead of ./gradlew for your current directory structure. I generally avoid gradle just due to versioning issues.

And apparently the problem was not with the plugin, but with the webp layer I was testing.. I checked with another webp layer I found on the internet.. it seems to be working fine. even without the changes you suggested. I would like to debug further as to what is going wrong with the layer I am trying.

I don't know then -- I wasn't seeing WebP when I was stepping through the code without the scanForPlugins bit.

About the suggestion regarding the Class-Path and the name, would this be a prerequisite for adding it into the main plugin repository?

No. Using ant, however, is (just due to how the JOSM CI works). If you want to do dual ant/gradle build scripts, see the ​mapillary or ​MapWithAI plugins for examples.

If you want a very basic release CI for GitHub actions, I created one for ​Lanes that you can use. I should look into making it a discrete repository so that changes can be more easily propagated. Right now, the only change you should have to make is to change the default branch name from master to main in the workflow (at the time I wrote the script, I was unable to use a variable for that).

Just keep in mind that someday you may want to add additional formats (specifically AVIF, and maybe JPEG XL), and then you would either have to refactor the plugin, add another plugin, or start fiddling with different build specifications.

[Sreeram K <kandimalla.sreeram@…>]

I will look into the options you provided and get ping back here once I am done. For now I have another problem/query.. the imagery layer I had problems with, which also happens to be the reason i attempted this plugin is hosted on Google Cloud Storage by me and looks like JOSM blacklisted all "*.googleapis.*".. this seems a bit extreme. Do you know anyway for me to override this?

[taylor.smock]

The *.googleapis.* is actually blacklisted by the OpenStreetMap API.

See ​https://api.openstreetmap.org/api/0.6/capabilities (<blacklist regex=".*\.google(apis)?\..*/.*"/>).

The only suggestion I have is to set up a domain that is not google related, and use that instead. We are not going to work around the OpenStreetMap blacklist.

With that being said, feel free to open a ticket on the ​openstreetmap-website repository, if you feel that the blacklist is overly broad. But I don't think it will go anywhere (just due to not wanting to chance copyright issues with Google Maps).

[Sreeram K <kandimalla.sreeram@…>]

Thanks for the pointers.. I will consider my options.

[Sreeram K <kandimalla.sreeram@…>]

Published the plugin to the plugin wiki page. Shows up as an external plugin in the plugin list. Has been tested on windows, linux and mac. Works nicely. In the end I had to put in the scanForPlugins line. Doesn't work consistently otherwise. Thanks for all the help @taylor.smock

[taylor.smock]

I'm going to go ahead and close this again since Sreeram released a plugin for webp.

[anonymous]

The Sreeram's webp plugin is does not work on Apple M1 or non-x86 architectures (the docs are wrong).

​https://github.com/haraldk/TwelveMonkeys as suggested earlier is a brilliant drop-in replacement for ImageIO with better compatibility and WebP support. It is also faster.

[kandimalla.sreeram@…]

When I last looked at it, TwelveMonkeys didn't have support full read support for WEBP, That's why I dropped the idea( I was the one who suggested it BTW :) ). Also, about the existing plugin not working on non-x86 Apple M1, I did a spot check on my laptop.. it worked. I am guessing my JVM might have been running in x86 compatibility mode. Looks like proper WEBP support is now available in TwelveMonkeys. Maybe you can submit a pull request to my repo or start a new plugin pulling in all of the image formats supported by TwelveMonkeys?

[Stereo]

Yeah, JOSM runs in x86 compatibility mode on M1 Macs, because automatic builds using github actions are complicated (​https://github.com/github/roadmap/issues/528) in the way we're currently building things.

On my homemade M1 build, the webp plugin installs, but adding ​https://stereo.dev.openstreetmap.org/testglobe.webp as a test tile layer produces an exception:

STACK TRACE

Thread: TMS-downloader-9 (116)
java.lang.NoClassDefFoundError: Could not initialize class com.luciad.imageio.webp.WebPDecoderOptions
	at com.luciad.imageio.webp.WebPReadParam.<init>(WebPReadParam.java:24)
	at com.luciad.imageio.webp.WebPReader.getDefaultReadParam(WebPReader.java:147)
	at java.desktop/javax.imageio.ImageIO.read(Unknown Source)
	at java.desktop/javax.imageio.ImageIO.read(Unknown Source)
	at org.openstreetmap.gui.jmapviewer.Tile.loadImage(Tile.java:247)
	at org.openstreetmap.josm.data.imagery.TMSCachedTileLoaderJob.tryLoadTileImage(TMSCachedTileLoaderJob.java:328)
	at org.openstreetmap.josm.data.imagery.TMSCachedTileLoaderJob.loadingFinished(TMSCachedTileLoaderJob.java:210)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.finishLoading(JCSCachedTileLoaderJob.java:265)
	at org.openstreetmap.josm.data.cache.JCSCachedTileLoaderJob.run(JCSCachedTileLoaderJob.java:234)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/1y/hwgwq23x3jbb7cyj0j998yrh0000gp/T/6605156055280924213libwebp-imageio.dylib [in thread "JCS Downloading: https://stereo.dev.openstreetmap.org/testglobe.webp"]
	at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
	at java.base/java.lang.Runtime.load0(Unknown Source)
	at java.base/java.lang.System.load(Unknown Source)
	at com.luciad.imageio.webp.NativeLibraryUtils.loadFromJar(WebP.java:131)
	at com.luciad.imageio.webp.WebP.loadNativeLibrary(WebP.java:31)
	at com.luciad.imageio.webp.WebP.<clinit>(WebP.java:38)
	at com.luciad.imageio.webp.WebPDecoderOptions.<clinit>(WebPDecoderOptions.java:20)
	... 12 more

[Stereo]

Reopening since there's only a partial solution.

[stoecker]

Instead of opening an old ticket here, open a ticket for the plugin at github where it can be fixed.

[taylor.smock]

In 36093/osm:

See #14361: Support for image/webp

[taylor.smock]

@stoecker: I'm not going to make a binary for the plugin I just pushed to svn yet -- I want to give you a chance to make certain that the download code isn't going to kill our servers (it downloads the ImageIO dependencies off of our nexus).

I don't think it will, but I'd rather not kill our servers accidentally.

[stoecker]

Static downloads aren't an issue for the server. ;-)

[taylor.smock]

Replying to stoecker:

Static downloads aren't an issue for the server. ;-)

Good to know. I didn't know if the downloads were static or generated on demand (I've seen some interesting implementations for stuff before).

[taylor.smock]

In 36094/osm:

See #14361 (dist): Support for image/webp